Pith. sign in

REVIEW 4 major objections 6 minor 26 references

Evaluating Generative Models for Tabular Data: Novel Metrics and Benchmarking

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that FAED, a Fréchet distance computed on autoencoder latents, detects quality decrease, mode drop, and mode collapse in tabular data where existing metrics fail.

desk verdict Three FID-style metrics for tabular data, but the experiments never train a generative model, so the central claim outruns the evidence. read the letter →

arxiv 2504.20900 v1 pith:4QYAGH32 submitted 2025-04-29 cs.LG

Dayananda Herurkar , Ahmad Ali , Andreas Dengel This is my paper · ORCID
classification cs.LG
keywords tabulardatagenerativemodelevaluationFréchetdistanceautoencodermodecollapsedropqualitydecreaseintrusiondetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that tabular-data generative models need their own evaluation metrics, and that three new metrics—FAED, FPCAD, and RFIS—meet that need better than existing scores. The central claim is that FAED, a Fréchet distance computed on the latent representations of a pre-trained autoencoder, reliably detects all three failure modes the authors inject into real data: noise-driven quality decrease, mode drop, and mode collapse. FPCAD, a PCA-based variant, detects some but behaves inconsistently across datasets; RFIS, an entropy-based score, works for noise and collapse but not mode drop. The support comes from perturbing three intrusion-detection datasets (UNSW-NB15, TON-IoT, CICIDS-2017), not from training generative models and scoring their outputs, so the paper's case rests on whether those perturbations faithfully stand in for real generative failure. If the claim holds, FAED would give practitioners a single quantitative signal for ranking tabular generators where current metrics give conflicting or flat answers.

What carries the argument

The central object is the Fréchet distance (Wasserstein-2 distance) between two multivariate Gaussians, Equation (1): $\lVert\mu_r - \mu_s\rVert^2 + \mathrm{Tr}(\Sigma_r + \Sigma_s - 2(\Sigma_r\Sigma_s)^{1/2})$, applied to feature vectors extracted by a pre-trained autoencoder for FAED or by PCA for FPCAD. RFIS replaces this with an entropy gap, $\mathbb{E}[H(p(y)) - H(p(y|x))]$, between the marginal and conditional label distributions from a random forest. The Fréchet terms do the detection work: they compare the location and spread of latent representations, so any perturbation that shifts the latent mean or inflates the covariance registers as a larger distance.

What would settle it

Train a generative model on one of the three datasets with a known, controlled weakness—for example, a GAN that produces only one class, or a diffusion model trained on a subset of modes—and compute FAED between real and generated rows. If FAED's relative score stays near the baseline or fails to rank the weakened model below a healthy one, the central claim that FAED detects real generative mode collapse is refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a distance computed in autoencoder latent space separates intact tabular data from data degraded in ways that mimic generator failures. FAED computes Equation (1), the Fréchet distance between the mean and covariance of latent vectors from real rows and from a candidate generator, so a lower value means a closer match. In the three intrusion-detection datasets, the relative FAED score moves sharply and monotonically with increasing noise, with successive removal of the most common value combinations, and with replacement of rows by mean/mode prototypes, while SDV Fidelity, TRTS, and TSTR stay flat or erratic. The paper concludes that FAED captures structural distortion that existing metrics overlook and proposes it as a general-purpose evaluation metric for tabular generative models; FPCAD and RFIS are presented as partial alternatives with known limitations.

Load-bearing premise

The load-bearing premise is that adding noise, dropping the most common value combinations, and replacing rows with mean and mode prototypes faithfully reproduces the ways real generative models fail; the paper never trains a GAN, VAE, or diffusion model to check this.

Editorial extensions

If this is right

  • FAED can serve as a single quantitative score for ranking tabular generative models, because it is the only tested metric that responds to all three injected failure modes.
  • Relying on SDV Fidelity or TRTS alone is risky: these scores can stay flat while the data has lost structure.
  • FPCAD provides a pre-training-free option where training an autoencoder is impractical, but its scores must be read dataset-by-dataset.
  • RFIS complements the Fréchet metrics for quality and collapse detection, but should not be trusted for mode drop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because FAED needs only a pre-trained autoencoder and a distance computation, it could be embedded in automated model-selection loops for tabular generators; the paper does not study that use, but the metric's cost is concentrated in the one-time autoencoder training.
  • FAED and FPCAD both assume Gaussian latent distributions, so their disagreement on datasets like CICIDS-2017 is a testable probe for when that approximation breaks; adding a non-parametric distance as a control would show whether the assumption is the culprit.
  • The injection protocol itself—noise, mode drop, mode collapse—could be reused as a standardized stress test for any future tabular evaluation metric, independent of whether FAED is adopted.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes three new evaluation metrics for tabular generative models: FAED (Fr\'echet distance on autoencoder latent representations), FPCAD (Fr\'echet distance on PCA projections), and RFIS (an entropy-based score using a random forest classifier). The authors evaluate these metrics against SDV Fidelity, TSTR, and TRTS on three intrusion-detection datasets by injecting three types of controlled corruption into real test data: Gaussian noise (quality decrease), removal of common value combinations (mode drop), and replacement of rows with mean/mode values (mode collapse). They report that FAED is the most sensitive to all injected corruptions, while FPCAD and RFIS show partial or inconsistent sensitivity. The paper claims that FAED 'effectively captures generative modeling issues overlooked by existing metrics.'

Significance. If the central claim were established, FAED would be a practical and computationally light diagnostic for tabular generative models, filling a real gap in the literature. The paper is also useful in identifying limitations of accuracy-based utility metrics like TRTS and in adapting FID/IS-style ideas to mixed-type tabular data. The proposed metrics are clearly defined and the comparative benchmark across three datasets is systematic. However, the current experimental evidence is restricted to hand-crafted distortions of real data and does not involve any generative model, so the significance as demonstrated is limited to a perturbation-sensitivity study rather than a validated evaluation framework for generative models.

major comments (4)
  1. [IV-B, IV-C, Eq. (6)] The experimental protocol never trains or samples a generative model. In Section IV-C, step 4 defines scoregen as the score obtained by comparing distorted real test data against the training set, not by comparing generated samples with real data. All 'generative modeling issues' in Section IV-B are hand-crafted corruptions of real rows (Gaussian noise, dropped value combinations, mean/mode replacement). Consequently, the results in Tables II and III demonstrate only that FAED, FPCAD, RFIS, and the baseline metrics respond to these specific perturbations; they do not establish that any metric can evaluate the output of a GAN, VAE, or diffusion model. Since the abstract and conclusion claim that FAED 'effectively captures generative modeling issues,' a bridging experiment with actual generative models is required.
  2. [IV-C, Eq. (6), Tables II-III] The validation is self-referential for distance-based metrics. FAED and FPCAD are Fr\'echet distances between distributional summaries; adding Gaussian noise or discarding rows changes the mean and covariance, so the distance necessarily increases. The paper does not compare FAED with a simple baseline such as the Fr\'echet distance computed on raw features, maximum mean discrepancy, or even a mean/covariance distance without an autoencoder. Without such controls, the sensitivity shown in Tables II and III is an expected property of any distributional distance and does not demonstrate that the autoencoder latent space adds diagnostic value. A control experiment or theoretical argument is needed to support the claim that FAED captures structural issues beyond trivial distribution shift.
  3. [III-B, Figure 1] The FPCAD procedure is not fully specified. Section III-B states that PCA reduces dimensionality to 'principal components that preserve the most significant variance' but does not state the number of components or a variance threshold, while Figure 1 says the first two principal components are extracted. These are contradictory instructions, and the ambiguity makes FPCAD irreproducible. The authors should fix the number of components, report sensitivity to that choice, and align the figure with the text.
  4. [V, Tables II-III] All reported values are point estimates without error bars, multiple seeds, or significance tests. The protocol includes randomness in the autoencoder training, PCA, random forest training, noise sampling, and train/test splitting, so the claimed consistent trends (e.g., 'FAED exhibits strong sensitivity') may not be statistically reliable. The paper should report means and standard deviations over repeated runs and test whether score changes are significant relative to baseline noise.
minor comments (6)
  1. [V-B, Table II] In Section V-B, the FAED single mode drop values for UNSW-NB15 are reported as -9.223 and -20.067, but Table II lists -8.289 and -21.241; the text and table should be reconciled.
  2. [Eq. (6), Tables II-III] Equation (6) uses a single formula for all metrics, but some metrics are higher-is-better and others lower-is-better; the 'relative score' therefore means opposite things depending on the metric. The arrows in Tables II and III partially address this, but the text should define the sign convention explicitly.
  3. [III-C] RFIS's entropy contrast depends on class balance; in the highly imbalanced intrusion datasets, p(y) is far from uniform, so the interpretation of 'diversity' via marginal entropy is confounded with class prior. Discuss or adjust.
  4. [IV-B] The Gaussian noise is described by standard deviation alpha=0.1..0.5, but no feature scaling or normalization is specified, so the noise level's meaning depends on each feature's scale. Report preprocessing and consider per-feature noise.
  5. [V, Figure 2] Figure 2 is missing from the manuscript; the text refers to it but only a summary paragraph appears. Insert the figure or refer to a table.
  6. [Footnote, Code availability] The footnote 'Code available' points to https://github.com/madhajj/Thesis Ali, which appears to be a personal repository rather than a project-specific release with versioning, dependencies, and license. Provide a proper repository or state the commit hash.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: FAED/FPCAD/RFIS are defined independently of the injected corruptions; the validation gap (no generative model trained) is an external-validity concern, not a constructional circularity.

full rationale

The paper's metrics are defined in Section III as distributional distances and entropy contrasts (Eqs. 1-2), and the 'generative modeling issues' in Section IV-B are operationalized as concrete perturbations of real test data. A Frechet distance will, by definition, increase when Gaussian noise inflates the covariance of one distribution or when mode-dropping shifts the mean and covariance, so the experiments are internally consistent sanity checks rather than circular predictions. The central claim that FAED 'effectively captures generative modeling issues' is not forced by any fitted parameter or by a self-citation chain; the paper invokes no load-bearing uniqueness theorem and does not rename an existing result as a new one. The principal weakness is a missing bridge: no GAN, VAE, or diffusion model is trained or sampled, so the hand-crafted corruptions are only a proxy for real generative failures. That is a threat to external validity and to the strength of the conclusion, but it is not circularity under the definitions used here, because the corruption protocol is not defined in terms of FAED and FAED is not fitted to the experimental outcomes. Accordingly, no circular step is identified and the score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central load-bearing assumptions are the Gaussian assumption for FAED and FPCAD, the shared PCA requirement for FPCAD, and above all the proxy assumption that perturbing real data mimics generative model failures. The free parameters listed are experimental design choices that bound the claims; the paper does not fit a theory but the claims are conditional on these hand-selected settings.

free parameters (3)
  • Noise standard deviation sweep values = alpha = 0.1 to 0.5
    Chosen by hand in Section IV-B to simulate quality decrease; the sensitivity claim is only demonstrated for these magnitudes.
  • Top-5 repeated combinations for successive mode drop = top 5 combinations
    Ad hoc choice in Section IV-B; results may depend on this number.
  • Bottom combination retention percentages = 10 to 50 percent
    Selected in Section IV-B for extreme mode drop; results depend on these thresholds.
assumptions (5)
  • domain assumption Latent feature representations of real and generated data follow a continuous multivariate Gaussian distribution.
    Equation (1), the Fréchet distance, is exact only for Gaussians. The authors note in Section III-A that this may not hold across diverse datasets.
  • ad hoc to paper Injecting noise, dropping common value combinations, and replacing rows with mean or mode values accurately simulates the failure modes of real generative models.
    The entire validation in Section IV-B depends on this proxy. No generative model is trained, so the connection is asserted rather than demonstrated.
  • domain assumption FPCAD applies a shared PCA mapping to both real and generated data.
    Section III-B says PCA is applied to both datasets but does not specify whether the transform is fit once and shared. If separate PCA transforms are used, the Fréchet distance between the two reduced spaces is not meaningful.
  • domain assumption Random forest predicted class probabilities are meaningful for RFIS on imbalanced intrusion detection data.
    RFIS in Section III-C assumes conditional label distributions have low entropy and marginal distributions are uniform, which may not hold for highly imbalanced binary data.
  • domain assumption The train-test split gap is a valid stand-in for the real versus generated gap.
    Section IV-C computes baseline scores by comparing the test set against the train set, treating that comparison as the highest achievable reference for a generative model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Generative Models for Tabular Data: Novel Metrics and Benchmarking." pith.science (2026). https://pith.science/paper/4QYAGH32

@misc{pith2026250420900,
  author       = {Pith},
  title        = {Pith review of: Evaluating Generative Models for Tabular Data: Novel Metrics and Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4QYAGH32}},
  note         = {Machine review of arXiv:2504.20900}
}
read the original abstract

Generative models have revolutionized multiple domains, yet their application to tabular data remains underexplored. Evaluating generative models for tabular data presents unique challenges due to structural complexity, large-scale variability, and mixed data types, making it difficult to intuitively capture intricate patterns. Existing evaluation metrics offer only partial insights, lacking a comprehensive measure of generative performance. To address this limitation, we propose three novel evaluation metrics: FAED, FPCAD, and RFIS. Our extensive experimental analysis, conducted on three standard network intrusion detection datasets, compares these metrics with established evaluation methods such as Fidelity, Utility, TSTR, and TRTS. Our results demonstrate that FAED effectively captures generative modeling issues overlooked by existing metrics. While FPCAD exhibits promising performance, further refinements are necessary to enhance its reliability. Our proposed framework provides a robust and practical approach for assessing generative models in tabular data applications.

Figures

Figures reproduced from arXiv: 2504.20900 by the authors.

Figure 1
Figure 1. Overview of the proposed evaluation metrics for generative models in the tabular data domain. (a) Fr [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Results of the quality decrease experiment, illus [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 11 canonical work pages

  1. [1]

    Improved techniques for training GANs,

    T. Salimans, I. Goodfellow, W. Zaremba, V . Cheung, A. Radford, and X. Chen, “Improved techniques for training GANs,” in Advances in Neural Information Processing Systems (NeurIPS) , vol. 29, 2016

    work page 2016

  2. [2]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017

    2017

  3. [4]

    Pros and cons of GAN evaluation measures,

    A. Borji, “Pros and cons of GAN evaluation measures,” Computer Vision and Image Understanding , vol. 179, pp. 41–65, 2019

    work page 2019

  4. [5]

    Tab-distillation: Impacts of dataset distillation on tabular data for outlier detection,

    D. Herurkar, F. Raue, and A. Dengel, “Tab-distillation: Impacts of dataset distillation on tabular data for outlier detection,” in Proceedings of the 5th ACM International Conference on AI in Finance , 2024, pp. 804–812

    work page 2024

  5. [6]

    Fin-fed- od: Federated outlier detection on financial tabular data,

    D. Herurkar, S. Palacio, A. Anwar, J. Hees, and A. Dengel, “Fin-fed- od: Federated outlier detection on financial tabular data,” arXiv preprint arXiv:2404.14933, 2024

    arXiv 2024

  6. [7]

    Recol: Reconstruction error columns for outlier detection,

    D. Herurkar, M. Meier, and J. Hees, “Recol: Reconstruction error columns for outlier detection,” in German Conference on Artificial Intelligence (K ¨unstliche Intelligenz). Springer, 2023, pp. 60–74

    work page 2023

  7. [8]

    Data synthesis based on generative adversarial networks,

    N. Park et al. , “Data synthesis based on generative adversarial networks,” Proceedings of the VLDB Endowment , vol. 11, no. 10, pp. 1071–1083, June 2018. [Online]. Available: http://dx.doi.org/10.14778/ 3231751.3231757

    arXiv 2018

  8. [10]

    Modeling tabular data using conditional gan,

    L. Xu et al. , “Modeling tabular data using conditional gan,” arXiv preprint , vol. arXiv:1907.00503, 2019. [Online]. Available: https://arxiv.org/abs/1907.00503

    arXiv 1907

Show all 26 references
  1. [11]

    The synthetic data vault,

    N. Patki, R. Wedge, and K. Veeramachaneni, “The synthetic data vault,” in 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA). IEEE, 2016, pp. 399–410

    2016

  2. [13]

    V AEM: A deep generative model for heterogeneous mixed type data,

    C. Ma et al. , “V AEM: A deep generative model for heterogeneous mixed type data,” arXiv preprint, vol. arXiv:2006.11941, 2020. [Online]. Available: https://arxiv.org/abs/2006.11941

    2006 arXiv

  3. [14]

    Available: https://arxiv.org/abs/1703.06490

    [Online]. Available: https://arxiv.org/abs/1703.06490

    arXiv

  4. [16]

    Tabddpm: Modelling tabular data with diffusion models,

    A. Kotelnikov et al., “Tabddpm: Modelling tabular data with diffusion models,” in Proceedings of the 40th International Conference on Ma- chine Learning (ICML) , 2023

    2023

  5. [17]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” 2020. [Online]. Available: https://arxiv.org/abs/2006.11239

    2020 arXiv

  6. [18]

    Photorealistic text-to-image diffusion models with deep language understanding,

    C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi, “Photorealistic text-to-image diffusion models with deep language understanding,” 2022. [Online]. Availabl...

    2022 arXiv

  7. [19]

    Progressive growing of gans for improved quality, stability, and variation,

    T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” 2018. [Online]. Available: https://arxiv.org/abs/1710.10196

    2018 arXiv

  8. [20]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” 2022. [Online]. Available: https://arxiv.org/abs/2112.10752

    2022 arXiv

  9. [21]

    Time-series generative adversarial networks,

    J. Yoon, D. Jarrett, and M. van der Schaar, “Time-series generative adversarial networks,” in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch ´e-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc.,

  10. [22]

    Hierarchical text-conditional image generation with clip latents,

    A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” 2022. [Online]. Available: https://arxiv.org/abs/2204.06125

    2022 arXiv

  11. [23]

    Predict, refine, synthesize: Self-guiding diffusion models for probabilistic time series forecasting,

    M. Kollovieh, A. F. Ansari, M. Bohlke-Schneider, J. Zschiegner, H. Wang, and Y . Wang, “Predict, refine, synthesize: Self-guiding diffusion models for probabilistic time series forecasting,” 2023. [Online]. Available: https://arxiv.org/abs/2307.11494

    2023 arXiv

  12. [24]

    Cross-domain transformation for outlier detection on tabular datasets,

    D. Herurkar, T. Sattarov, J. Hees, S. Palacio, F. Raue, and A. Dengel, “Cross-domain transformation for outlier detection on tabular datasets,” in International Joint Conference on Neural Networks, IJCNN 2023, Gold Coast, Australia, June 18-23, 2023. IEEE, 2023, pp. 1–8. [Onli...

    2023

  13. [25]

    Explaining anomalies us- ing denoising autoencoders for financial tabular data,

    T. Sattarov, D. Herurkar, and J. Hees, “Explaining anomalies us- ing denoising autoencoders for financial tabular data,” arXiv preprint arXiv:2209.10658, 2022

    2022 arXiv

  14. [26]

    Real-valued (medical) time series generation with recurrent conditional gans,

    C. Esteban, S. L. Hyland, and G. R ¨atsch, “Real-valued (medical) time series generation with recurrent conditional gans,” 2017. [Online]. Available: https://arxiv.org/abs/1706.02633

    2017 arXiv

  15. [30]

    [Online]

    Synthetic Data Metrics , DataCebo, Inc., 9 2022, v0.7.0. [Online]. Available: https://docs.sdv.dev/sdmetrics/

    2022

  16. [2018]

    Available: https://arxiv.org/abs/1811.11264

    [Online]. Available: https://arxiv.org/abs/1811.11264

    arXiv

  17. [2019]

    Available: https://proceedings.neurips.cc/paper files/ paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf

    [Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf

    2019

  18. [2023]

    Available: https://arxiv.org/abs/2309.01472

    [Online]. Available: https://arxiv.org/abs/2309.01472

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.