Pith. sign in

REVIEW 3 major objections 4 minor 38 references

Common metrics certify synthetic data that have lost all inter-column dependence; a new test reveals a real gap that 16x capacity cannot close.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 14:56 UTC pith:2CMHAD3B

load-bearing objection A clean diagnostic that exposes a real blind spot in tabular-generation metrics; the causal story about why the gap persists is shakier than the abstract suggests. the 3 major comments →

arxiv 2607.21636 v2 pith:2CMHAD3B submitted 2026-07-20 cs.LG cs.AI

Measuring the Dependency Gap: Diagnosing Inter-Column Fidelity in Tabular Generative Models

classification cs.LG cs.AI
keywords synthetic tabular datainter-column dependencyclassifier two-sample testXGB-C2STflow matchingdiffusion modelsfidelity metricsminority-class utility
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the metrics most used to certify synthetic tabular data—logistic-regression C2ST and pairwise Trend scores—are blind to inter-column dependency, so a trivial fully-factorized baseline that destroys all joint structure is judged nearly indistinguishable from real data. It introduces a diagnostic that decomposes a strong gradient-boosted classifier two-sample test into marginal, dependency, and numerical-categorical cross components, anchored between a fully-factorized lower bound and a real-data oracle. Applied to state-of-the-art flow-matching and diffusion generators, the diagnostic finds a significant dependency gap on four datasets, which tracks minority-class utility. The paper presents evidence that this gap is not a structural limitation of mean-field objectives (a recovery theorem and a controlled XOR experiment), and that it is not closed by a 16x capacity increase, an in-model dependency module, or post-hoc copula correction—pointing to the objective's lack of direct dependency supervision as the cause.

Core claim

The central claim is that state-of-the-art flow-matching (EF-VFM/TabbyFlow) and diffusion (TabDiff) tabular generators leave a measurable dependency gap—synthetic rows have correct marginals and even matched second-order moments, yet the joint distribution is still distinguishable from real data by an interaction-sensitive tree classifier. The paper shows this gap is significant (dependency component of +0.05 to +0.08 AUC over chance), is correlated with minority-class utility loss, and is not attributable to the factorized 'mean-field' architecture: a cited recovery theorem shows such objectives are asymptotically exact, and a controlled XOR experiment shows independent categorical decoding

What carries the argument

The central object is the XGB-C2ST decomposition: a two-sample test using gradient-boosted trees (an interaction-sensitive learner) as the discriminator, whose AUC is split into a marginal component (computed after independently permuting each column's rows, destroying cross-column dependence) and a dependency component (the difference between full and marginal discriminability), plus a numerical-categorical cross subcomponent isolated by block-shuffling. The decomposition is anchored by a fully-factorized reference (correct marginals, zero dependency) as a lower bound and a held-out real-data subsample as an oracle upper bound, so that a generator's dependency fidelity is read relative to t

Load-bearing premise

The conclusion that the dependency gap is not a structural limitation of mean-field objectives rests on an external recovery theorem for the infinite-capacity limit, which the paper neither reproduces nor verifies for the finite trained models it tests—if that theorem does not apply to them, the gap could still be a representational limit.

What would settle it

Train EF-VFM or TabDiff with an objective that explicitly supervises joint dependence (e.g., adding a divergence term on the generated joint distribution) using the same data and pipeline; if the XGB-C2ST dependency component drops to the oracle level, the paper's claim that the gap is caused by missing dependency supervision is supported—but if the gap persists despite direct joint supervision, the 'no structural blind spot' conclusion is false. Alternatively, measure the trained model's velocity field against the mean-field optimum on a controlled problem; a mismatch would reopen the structu

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners using logistic-regression C2ST or pairwise Trend to certify synthetic data will certify a generator that has destroyed all inter-column dependence; the paper's diagnostic should be used instead.
  • Dependency loss is not a cosmetic flaw: destroying dependency collapses minority-class F1 (e.g., 0.66 to 0.10 on adult), so any fidelity report that omits the dependency axis can mislead downstream fairness and risk applications.
  • The measured gap is a property of the objective, not the architecture, so simply scaling up existing mean-field flow-matching or diffusion models will not close it; method design should focus on objectives that directly supervise joint structure.
  • Neither covariance-matching fixes nor post-hoc copula correction can close the gap, because the residual mismatch lives in higher-order structure—the generator already matches second-order cross-moments.
  • The gap replicates across two generator families (flow matching and diffusion), so the conclusion generalizes beyond a single model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The diagnostic could be applied to other tabular generator families (e.g., GAN- and LLM-based synthesizers) to test whether they share the dependency gap; the paper's mechanism predicts yes, since their objectives also factor over columns.
  • The paper's 'no cheap intervention' result suggests that closing the gap will require a non-trivial change to the training signal—for instance, a term that directly rewards joint fidelity. A direct test would be to train with such a supervision term and observe whether the dependency component drops toward the oracle; the paper leaves this as an open question.
  • The recovery theorem's infinite-capacity assumption is the hinge: if finite-trained models do not operate near the mean-field optimum, the structural-blind-spot explanation (H1) is not fully ruled out. A practical check would be to measure how close the trained velocity field is to the mean-field solution on a small synthetic problem.
  • The XOR control suggests that independent decoding is not the bottleneck, so interventions at the output head (e.g., a cross-coupling decoder) are unlikely to help; the bottleneck is earlier, in the training signal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces a diagnostic for inter-column dependency fidelity in tabular generative models. A gradient-boosted-tree classifier two-sample test (XGB-C2ST) is decomposed into marginal, dependency, and numerical–categorical cross components by independent row permutations and block permutations, with a fully-factorized lower bound and a real-data oracle upper bound. Applying this diagnostic to TabbyFlow/EF-VFM and TabDiff on adult, default, bank, magic, and shoppers, the paper reports that a fully-factorized baseline that destroys all dependency is scored as near-indistinguishable by logistic-regression C2ST, while XGB-C2ST exposes a significant dependency gap in both generators. The paper further argues that the gap is not a structural blind spot of mean-field flow matching, is not closed by a 16x capacity increase, an explicit in-model dependency mechanism, or post-hoc copula correction, and concludes that the deficit points to the absence of direct dependency supervision in the training objective.

Significance. If the diagnostic is adopted, it gives practitioners a concrete way to detect a failure mode that standard fidelity metrics miss, and the empirical demonstration that LR-C2ST certifies a zero-dependency baseline as faithful is a valuable cautionary result. The protocol is careful: held-out test sets, fixed seeds, five-seed statistics, paired tests, confidence intervals, disclosed exclusions, and released code. The paper also contains honest negative results and a controlled XOR check. The dependency diagnostic itself is simple, interpretable, and anchored by external references rather than fitted residuals. However, the causal interpretation—that the gap is due to the absence of direct dependency supervision rather than a capacity or structural limit—rests on an external recovery theorem that is not verified on the finite-trained models, and on negative results whose strength varies by dataset. The empirical core is solid; the explanatory claim is currently supported more by elimination than by direct evidence.

major comments (3)
  1. [§5 'No structural blind spot'; abstract] The paper rules out H1 (structural blind spot) by citing Nasution et al. (2026), which is an infinite-capacity recovery result about the mean-field optimum. Section 5 then concludes that 'any observed gap must be attributed to finite capacity, optimization, or sampling discretization rather than to the objective’s form.' The finite-trained EF-VFM/TabDiff models are not shown to satisfy the theorem’s assumptions: affine interpolation in the one-hot endpoint is asserted rather than verified, the trained models are not shown to be at the mean-field optimum, and no comparison is made between the learned per-dimension posterior means and the ground-truth E[x1|x_t]. The capacity sweep in §5 tests width, not optimality, and §7 concedes that broader under-optimization and sampling discretization are entangled with H3. As a result, the abstract’s claim that the gap points 'to the absence of direc
  2. [§6 'Can In-Model Fixes Close It?'] The claim that post-hoc copula correction does not close the gap is one of the headline negative results in the abstract, but the manuscript does not report the copula experiment. The text says 'the empirical variant, in a prior controlled study under the same protocol' without a citation or results, and states that a Gaussian copula sits at the XGB-C2ST ceiling. This is not reproducible from the manuscript. Please provide the protocol, datasets, copula variants, tuning procedure, and the resulting XGB-C2ST/dependency/utility numbers, or remove/qualify this negative result. As written, one of the three main 'no cheap intervention' claims is unsupported in the manuscript itself.
  3. [§5 'Ruling out insufficient capacity (H2)'] The conclusion that a 16x capacity increase does not close the gap is tight only on adult, where the paired 95% CI is [-0.002, +0.019]. On bank and magic the intervals are wide ([-0.013, +0.022] and [-0.027, +0.007]), so reductions of roughly 0.01–0.03 in the dependency component cannot be excluded, and default—the dataset with the largest dependency gap (+0.077)—is excluded from the sweep because training is not cleanly optimized at any width. The paper’s own caveats are honest, but the abstract and contribution list state the capacity negative as a general result. Please restrict the claim to the datasets where the sweep is actually informative, or augment the sweep (for example, with additional widths/seeds or a direct measure of capacity utilization) so that the H2 elimination has comparable strength across all datasets.
minor comments (4)
  1. [§3 'Reference points' and 'Protocol and discipline'] There is a contradiction in the definition of the oracle: 'Reference points' says the oracle is 'a fresh subsample of real training data,' while the protocol says 'The real reference is always the held-out test set, never a set used to fit a reference.' Please clarify which real split is used for the oracle and confirm that no circularity is introduced.
  2. [§4 'Lost dependency degrades minority-class utility'] The generator-versus-oracle minority-class F1 shortfalls (0.012–0.046) are reported as point estimates without standard errors or confidence intervals, although the rest of the paper is statistically careful. Please report uncertainty for these utility comparisons, since the claim that the residual cost is 'consistent' depends on them.
  3. [§6] The 'sampler-level check' used to verify that the added cross-coupling mechanism actually changes generated samples is mentioned but not described. Please specify how this check is performed, as it is important for interpreting the null result.
  4. [§4 and §5] The one-sample t-tests for the dependency component are performed per dataset without multiple-testing correction. With p = 0.02 for magic, a simple Bonferroni correction across the four quantitative datasets would make this result not significant. Consider reporting adjusted p-values or explicitly labeling these as per-dataset exploratory tests.

Circularity Check

0 steps flagged

No significant circularity: the diagnostic is an anchored definition, the causal claims are explicitly hedged, and the load-bearing recovery theorem is external to the authors.

full rationale

The paper's central diagnostic is an operational definition rather than a fitted residual: dep = full - marg, where full and marg are XGB-C2ST AUCs computed on real vs. generated samples and on independently row-permuted versions of those samples. The fully-factorized lower bound (FF) and real-data oracle are constructed from the data and the generator's output, not fitted to produce the conclusion, so the blindness result is not forced by construction. The 16x capacity sweep is a genuine null result with paired confidence intervals, and the paper explicitly excludes default from that comparison because the matched-optimization premise could not be verified. The no-structural-blind-spot claim rests on an external recovery theorem by Nasution et al. (2026), with no author overlap with the present paper, so this is not a self-citation chain; failure to reproduce or verify the theorem's assumptions is a correctness/verification risk, not circularity. The paper is also unusually explicit about the epistemic status of its central explanation: H3 is 'supported by elimination and by consistency with Section 6, not by a direct intervention that adds dependency supervision and closes the gap,' and it concedes that 'broader under-optimization ... and sampling discretization are entangled with H3 and are not separately excluded.' No prediction in the paper is obtained by renaming a fitted input, and no equation reduces to its own input. Therefore the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical entities are postulated. The 'dependency component' is a defined measurement, not an entity. The central measurement has no fitted target parameters; the main load-bearing external input is the Nasution et al. asymptotic recovery theorem, plus standard evaluation assumptions about held-out sets and paired statistics. The free parameters listed are model/discriminator configurations and validation-tuned copula settings that set the scale of the reported numbers.

free parameters (3)
  • XGB-C2ST discriminator hyperparameters = 300 trees, depth 6 (default); sensitivity sweep 100/300/600 trees at depth 3/6/10
    The measured dependency magnitude grows with discriminator strength; the paper reads all values against same-discriminator FF/oracle anchors. This is a measurement configuration, not a scientific fit, but it sets the scale of the reported gap.
  • Model capacity width sweep = dim_t in {1024, 2048, 4096} (about 10.6M/42M/168M parameters)
    The conclusion '16x capacity does not close the gap' depends on these chosen widths and the matched-optimization assumption; widths are hand-picked, not derived.
  • Post-hoc copula tuning on validation = not reported
    The paper states copula variants were tuned on validation but does not give selected values or a table; this weakens the reported negative copula result.
axioms (5)
  • domain assumption The mean-field optimum of variational flow matching with affine interpolation recovers per-dimension posterior means and the induced deterministic flow transports to the true joint in the infinite-capacity limit (Nasution et al., 2026).
    Used to rule out H1 in Section 5 ('No structural blind spot'); the paper does not prove or verify its conditions for the trained EF-VFM/TabDiff models.
  • domain assumption The additive decomposition in AUC space: c2st(full) - c2st(permuted) isolates the contribution of inter-column dependency to class separability.
    The paper defines dep = full - marg in Section 3; this is meaningful only if permutation preserves marginals and if AUC differences are a valid proxy for distributional component contributions. The paper notes the share is a 'discriminability share' in Section 7.
  • domain assumption The real-data oracle (fresh subsample of training data) is an upper bound on achievable XGB-C2ST fidelity and a fair reference for dependency.
    Oracle anchors assume a fresh real subsample is effectively indistinguishable from the held-out test set (XGB-C2ST near 0.5) and that a generator may approach this. Used throughout Tables 1-2.
  • domain assumption Held-out test set rows are independent; train-on-synthetic/evaluate-on-real measures downstream utility.
    Standard evaluation assumption, invoked in Section 3; not separately verified for class imbalance.
  • domain assumption Matched-seed paired t-tests remove shared initialization/sampling noise sufficiently to support within-noise conclusions.
    Used in Sections 5-6 capacity/fix comparisons; the paper itself notes trajectories diverge and sign counts are screening only.

pith-pipeline@v1.3.0-alltime-deepseek · 12216 in / 16881 out tokens · 212575 ms · 2026-08-01T14:56:32.442713+00:00 · methodology

0 comments
read the original abstract

Synthetic tabular data is prized for preserving not just each column's marginal distribution but the dependencies between columns - structure that carries much of the discriminative signal for minority classes in imbalanced domains such as fraud detection and clinical risk. Yet the metrics most commonly used to certify such data are largely blind to this structure: a fully-factorized baseline that destroys all inter-column dependency is judged nearly indistinguishable from real data by the logistic-regression C2ST, while the pairwise Trend score is only partially sensitive. We introduce a dependency-aware fidelity diagnostic that decomposes a strong gradient-boosted classifier two-sample test (XGB-C2ST) into marginal, dependency, and numerical-categorical cross components, anchored between a fully-factorized lower bound and a real-data oracle upper bound. Applying it to state-of-the-art flow-matching (TabbyFlow/EF-VFM) and diffusion (TabDiff) generators, we expose a real dependency gap that standard metrics miss and that tracks minority-class utility. The gap is not a structural limitation of mean-field objectives - consistent with recent recovery results, these are asymptotically exact - yet it is stubborn: neither a 16x capacity increase nor post-hoc copula correction closes it, pointing to the absence of direct dependency supervision rather than a capacity or structural limit.

Figures

Figures reproduced from arXiv: 2607.21636 by Jie Zhang.

Figure 1
Figure 1. Figure 1: The XGB-C2ST gap above chance, decom￾posed into marginal and dependency segments. Error bars: ±1 std over seeds (five; three for the magic an￾chors); “Tabby” abbreviates TabbyFlow. FF (zero dependency) is almost all dependency, the oracle is ≈ 0, and TabbyFlow carries a substantial depen￾dency segment. ADULT SHOPPERS BANK MAGIC 0.0 0.2 0.4 0.6 0.8 1.0 score of the zero-dependency FF reference Trend (1.0 = … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 2 linked inside Pith

  1. [1]

    International Conference on Machine Learning (ICML) , year =

    Exponential Family Variational Flow Matching for Tabular Data Generation , author =. International Conference on Machine Learning (ICML) , year =

  2. [2]

    Transactions on Machine Learning Research (TMLR) , year =

    Flow Matching for Tabular Data Synthesis , author =. Transactions on Machine Learning Research (TMLR) , year =

  3. [3]

    arXiv preprint arXiv:2603.23016 , year =

    A Sobering Look at Tabular Data Generation via Probabilistic Circuits , author =. arXiv preprint arXiv:2603.23016 , year =

  4. [4]

    International Conference on Learning Representations (ICLR) , year =

    Revisiting Classifier Two-Sample Tests , author =. International Conference on Learning Representations (ICLR) , year =

  5. [5]

    International Conference on Learning Representations (ICLR) , year =

    TabDiff: A Mixed-type Diffusion Model for Tabular Data Generation , author =. International Conference on Learning Representations (ICLR) , year =

  6. [6]

    The Twelfth International Conference on Learning Representations (

    Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space , author =. The Twelfth International Conference on Learning Representations (. 2024 , note =

  7. [7]

    arXiv preprint arXiv:2601.22816 , year =

    Cascaded Flow Matching for Heterogeneous Tabular Data with Mixed-Type Features , author =. arXiv preprint arXiv:2601.22816 , year =

  8. [8]

    arXiv preprint arXiv:2602.13813 , year =

    Pawsterior: Variational Flow Matching for Structured Simulation-Based Inference , author =. arXiv preprint arXiv:2602.13813 , year =

  9. [9]

    International Conference on Learning Representations (ICLR) , year =

    Diffusion and Flow-based Copulas: Forgetting and Remembering Dependencies , author =. International Conference on Learning Representations (ICLR) , year =

  10. [10]

    Advances in Neural Information Processing Systems 33 (

    Jonathan Ho and Ajay Jain and Pieter Abbeel , title =. Advances in Neural Information Processing Systems 33 (. 2020 , note =

  11. [11]

    Score-Based Generative Modeling through Stochastic Differential Equations , booktitle =

    Yang Song and Jascha Sohl. Score-Based Generative Modeling through Stochastic Differential Equations , booktitle =. 2021 , note =

  12. [12]

    Yaron Lipman and Ricky T. Q. Chen and Heli Ben. Flow Matching for Generative Modeling , booktitle =. 2023 , note =

  13. [13]

    Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport , journal =

    Alexander Tong and Kilian Fatras and Nikolay Malkin and Guillaume Huguet and Yanlei Zhang and Jarrid Rector. Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport , journal =. 2024 , note =

  14. [14]

    Modeling Tabular data using Conditional

    Lei Xu and Maria Skoularidou and Alfredo Cuesta. Modeling Tabular data using Conditional. Advances in Neural Information Processing Systems 32 (. 2019 , note =

  15. [15]

    Chen , title =

    Zilong Zhao and Aditya Kunar and Robert Birke and Lydia Y. Chen , title =. Asian Conference on Machine Learning (. 2021 , note =

  16. [16]

    International Conference on Machine Learning (

    Akim Kotelnikov and Dmitry Baranchuk and Ivan Rubachev and Artem Babenko , title =. International Conference on Machine Learning (. 2023 , note =

  17. [17]

    International Conference on Learning Representations (

    Jayoung Kim and Chaejeong Lee and Noseong Park , title =. International Conference on Learning Representations (. 2023 , note =

  18. [18]

    International Conference on Machine Learning (

    Chaejeong Lee and Jayoung Kim and Noseong Park , title =. International Conference on Machine Learning (. 2023 , note =

  19. [19]

    Language Models are Realistic Tabular Data Generators , booktitle =

    Vadim Borisov and Kathrin Se. Language Models are Realistic Tabular Data Generators , booktitle =. 2023 , note =

  20. [20]

    International Conference on Learning Representations (

    Tennison Liu and Zhaozhi Qian and Jeroen Berrevoets and Mihaela van der Schaar , title =. International Conference on Learning Representations (

  21. [21]

    2016 , doi =

    Neha Patki and Roy Wedge and Kalyan Veeramachaneni , title =. 2016 , doi =

  22. [22]

    Frontiers in Big Data , volume =

    Michael Platzer and Thomas Reutterer , title =. Frontiers in Big Data , volume =. 2021 , doi =

  23. [23]

    Deep Neural Networks and Tabular Data:

    Vadim Borisov and Tobias Leemann and Kathrin Se. Deep Neural Networks and Tabular Data:. 2024 , doi =

  24. [24]

    International Conference on Learning Representations (

    James Jordon and Jinsung Yoon and Mihaela van der Schaar , title =. International Conference on Learning Representations (

  25. [25]

    Proceedings of the 22nd

    Tianqi Chen and Carlos Guestrin , title =. Proceedings of the 22nd. 2016 , doi =

  26. [26]

    Why do tree-based models still outperform deep learning on typical tabular data? , booktitle =

    L. Why do tree-based models still outperform deep learning on typical tabular data? , booktitle =. 2022 , note =

  27. [27]

    Chawla and Kevin W

    Nitesh V. Chawla and Kevin W. Bowyer and Lawrence O. Hall and W. Philip Kegelmeyer , title =. Journal of Artificial Intelligence Research , volume =. 2002 , doi =

  28. [28]

    Garcia , title =

    Haibo He and Edwardo A. Garcia , title =. 2009 , doi =

  29. [29]

    Nelsen , title =

    Roger B. Nelsen , title =. 2006 , isbn =

  30. [30]

    Insurance: Mathematics and Economics , volume =

    Kjersti Aas and Claudia Czado and Arnoldo Frigessi and Henrik Bakken , title =. Insurance: Mathematics and Economics , volume =. 2009 , doi =

  31. [31]

    2017 , institution =

    Dheeru Dua and Casey Graff , title =. 2017 , institution =

  32. [32]

    A Data-Driven Approach to Predict the Success of Bank Telemarketing , journal =

    S. A Data-Driven Approach to Predict the Success of Bank Telemarketing , journal =. 2014 , doi =

  33. [33]

    Borgwardt and Malte J

    Arthur Gretton and Karsten M. Borgwardt and Malte J. Rasch and Bernhard Sch. A Kernel Two-Sample Test , journal =. 2012 , url =

  34. [34]

    Cohen and Adrian Weller , title =

    James Jordon and Lukasz Szpruch and Florimond Houssiau and Mirko Bottarelli and Giovanni Cherubin and Carsten Maple and Samuel N. Cohen and Adrian Weller , title =. 2022 , eprint =

  35. [35]

    Neurocomputing , volume =

    Mikel Hernandez and Gorka Epelde and Ane Alberdi and Rodrigo Cilla and Debbie Rankin , title =. Neurocomputing , volume =. 2022 , doi =

  36. [36]

    Real-valued (Medical) Time Series Generation with Recurrent Conditional

    Crist. Real-valued (Medical) Time Series Generation with Recurrent Conditional. 2017 , eprint =

  37. [37]

    Using Generative Adversarial Networks for Improving Classification Effectiveness in Credit Card Fraud Detection , journal =

    Ugo Fiore and Alfredo. Using Generative Adversarial Networks for Improving Classification Effectiveness in Credit Card Fraud Detection , journal =. 2019 , doi =

  38. [38]

    arXiv preprint arXiv:2503.02161 , year =

    LLM-TabLogic: Preserving Inter-Column Logical Relationships in Synthetic Tabular Data via Prompt-Guided Latent Diffusion , author =. arXiv preprint arXiv:2503.02161 , year =