REVIEW 4 major objections 4 minor 42 references
A Comparative Study of Open-Source Libraries for Synthetic Tabular Data Generation: SDV vs. SynthCity
T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read In a six-way comparison on 1,000 energy-data rows, Synthcity's Bayesian Network reproduced the real tables most faithfully, while SDV's TVAE led when asked to generate 10,000 rows.
desk verdict Useful head-to-head benchmark, but the missing target variable and sampling procedure make the headline ranking unverifiable as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the evaluation protocol rather than a single generative model. Statistical similarity is a column-wise composite score that averages normalized differences in mean, median, and standard deviation, plus the Kolmogorov–Smirnov statistic and Wasserstein distance for numeric columns and mode agreement for categorical columns. Predictive utility uses the Train-on-Synthetic, Test-on-Real (TSTR) paradigm: four regression models (XGBRegressor, Random Forest Regressor, Support Vector Regressor, and Linear Regression) are trained on each synthetic dataset and tested on real data, then scored relative to the same models trained on real data. That scoring protocol is what carries all the paper's rankings.
What would settle it
Re-run the TSTR evaluation on the same energy dataset with the regression target explicitly named and with the 1,000 training rows drawn by stratified random sampling. If SDV's TVAE no longer produces the best 1:10 predictive utility score, or if Synthcity's Bayesian Network keeps that lead, the paper's headline ranking reverses.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is a ranking: Synthcity's Bayesian Network produces tables most statistically like the real data in both the 1:1 setting (fidelity score 96.53 out of 100) and the 1:10 setting (78.25), while SDV's TVAE produces synthetic rows with the highest predictive utility when expanding from 1,000 to 10,000 rows (score 0.31), where most other generators score near or below zero. The paper also finds that statistical similarity stays acceptably high across all generators as output scales, but predictive utility falls sharply, and that the two libraries' shared models behave differently, with SDV's CTGAN and TVAE outperforming their Synthcity counterparts in the 1:10 predictive test.
Load-bearing premise
The predictive-utility ranking rests on an unstated premise: that the regression target column among the dataset's 29 features and the way the 1,000 training rows were sampled are exactly what the evaluator assumed, but the paper never names the target or the sampling procedure.
Editorial extensions
If this is right
- A practitioner who needs statistically plausible tables from a small sample should reach for Synthcity's Bayesian Network before trying deep generators.
- A practitioner who needs to produce ten times more rows than they have and wants those rows to preserve predictive value should reach for SDV's TVAE.
- Generating at a 1:10 scale appears safe for distributional shape (fidelity scores 70–80) but risky for downstream modeling (most predictive-utility scores are negative), so the scale ratio is a decision variable, not a default.
- Because the paper finds no overall performance winner, library choice can reasonably be driven by documentation and community support, where SDV is judged stronger.
Reading between the lines
- Editorial inference: the Bayesian Network's dominance likely reflects the dataset's strong inter-column dependencies (energy consumption with weather and house sensors), so the ranking may not transfer to datasets with weak or nonlinear dependencies; a useful next test is repeating the protocol on a high-cardinality categorical dataset.
- Editorial inference: the TSTR scoring's sensitivity to the unstated regression target means the predictive-utility ranking is less robust than the fidelity ranking; re-running with the target column declared and stratified sampling of the 1,000 training rows is a cheap falsification.
- Editorial inference: the sharp drop in predictive utility at 1:10 while fidelity stays moderate suggests the fidelity metric rewards matching marginal distributions but fails to capture joint dependencies that matter for prediction; this motivates adding a dependence-preservation metric.
- Editorial inference: a practical extension would be to test whether training the generators on 2,000 or 5,000 rows changes the 1:10 ranking, since the paper's 1,000-row sample may understate deep models' behavior with more data.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript compares six tabular synthetic data generators drawn from two open-source libraries: Gaussian Copula, CTGAN, and TVAE from SDV, and Bayesian Network, CTGAN, and TVAE from Synthicity. The experiments use a UCI energy-consumption dataset, train each generator on 1,000 real rows, and generate synthetic data at 1:1 (1,000 rows) and 1:10 (10,000 rows) ratios. The generators are evaluated on statistical similarity and on predictive utility via a Train on Synthetic, Test on Real (TSTR) protocol with four regression models. The reported results are that Synthicity's Bayesian Network achieves the highest statistical fidelity in both scenarios, that SDV's TVAE performs best for predictive utility in the 1:10 setting, and that neither library is an overall performance winner, with SDV preferred on documentation and usability grounds.
Significance. If the claims held as stated, the paper would provide useful practical guidance for practitioners choosing between SDV and Synthicity. The tabulated results show internal consistency: the predictive-utility scores in Tables 5 and 6 are plausibly derived from the normalized MAE, MSE, and R2 differences in Tables 3 and 4, and the TSTR design is methodologically appropriate. However, the manuscript omits several load-bearing experimental details: the regression target column is not identified, the sampling of the 1,000-row training subset is not described, the weights of the custom statistical-similarity metric are not given, and no repository URL is provided. As written, the headline rankings are not reproducible from the paper, so the practical guidance cannot currently be verified.
major comments (4)
- [Methods: Comparison Process and Methods] The predictive-utility evaluation never states which of the 29 columns of the UCI energy dataset is the regression target, nor how the 1,000-row training subset was sampled (random, stratified, chronological, or fixed by a seed). Tables 5 and 6 are normalized differences relative to models trained on real data, so both choices affect every entry; the claim that TVAE (SDV) is the best 1:10 model is therefore not verifiable from the text as written.
- [Methods: Comparison Process and Methods] The statistical-similarity score is defined only as 'a weighted combination' of normalized mean/median/std, Kolmogorov-Smirnov, and Wasserstein differences, with no weights or aggregation details given. Tables 1 and 2, which support the headline claim that Synthicity's Bayesian Network has the highest fidelity in both settings, are rankings of this unspecified score; the weights are free parameters that should be reported.
- [Methods: Comparison Process and Methods] The text states that 'multiple evaluation rounds were conducted using repeated holdout validation' but gives no number of rounds, train/test split ratio, seeds, or number of synthetic-generation runs; it also gives no hyperparameters or package versions, and the 'accompanying GitHub repository' is not identified by URL. The SDV-versus-Synthcity differences in Tables 3 and 4 could be within stochastic variation, so the results cannot be distinguished from noise or independently checked as reported.
- [Results: Table 4 note] The note says that 'different amounts of real data were used for training' in the 1:1 and 1:10 experiments, but the paper does not state whether the 10,000 real training rows in the 1:10 condition are a superset of the 1,000 rows used to train the generators or an independent draw. This is essential for interpreting the comparison between models trained on 1:10 synthetic data and models trained on real data.
minor comments (4)
- [Methods: Data Generators Description] The section says 'four different Data Generators' although six models are evaluated; please clarify that four architectures are instantiated by two libraries.
- [Literature Review] The literature review describes statistical similarity as using 'reliable statistical functions provided by SDV', while the Methods and Abstract describe a 'custom' function; specify which components come from SDV and which are custom, since using a library under comparison inside the evaluation metric is a potential fairness concern.
- [Discussion & Conclusion] The phrase 'no significant distinction' and 'no significant performance gap' are used without reporting any significance test; please either provide a test or replace 'significant' with a non-statistical term such as 'substantive'.
- [General] The title uses 'SynthCity' while the body uses 'Synthicity' inconsistently; standardize the spelling for consistency.
Circularity Check
No significant circularity: the benchmark conclusions are evaluated against an external real-world dataset and do not reduce to their inputs by construction.
full rationale
This paper does not attempt a theoretical derivation; it reports an empirical benchmark of six synthetic-data generators against the UCI energy dataset, an external real-world source. The statistical-similarity and predictive-utility scores are computed from held-out real data via standard metrics (KS statistic, Wasserstein distance, MAE/MSE/R2 in a TSTR setup), so the central claims are falsifiable by re-running the comparison rather than being forced by definition. The only mild self-reference is that some statistical functions are 'provided by SDV,' one of the libraries under comparison, but the paper describes the scoring as a custom column-by-column computation of standard statistics, and using a library's implementation of generic statistical tests is a fairness or implementation concern, not a circular reduction of the conclusion to its input. Similarly, the unstated regression target and 1,000-row sampling procedure are reproducibility risks that make the predictive-utility ranking hard to verify, but they do not constitute a self-definitional or fitted-input-called-prediction step. There are no uniqueness theorems imported from the authors, no ansatz smuggled in via self-citation, and no renaming of a known result; the conclusions are presented as measurements, not as implications that reproduce their own premises. Accordingly, no circularity step meets the evidentiary bar, and the appropriate score is 0.
Assumptions & free parameters
free parameters (1)
- Column-score weights in statistical similarity function =
not reported
assumptions (6)
- standard math Sklar's theorem (Eq. 1) and Bayesian network factorization (Eq. 2) are valid for the data.
- standard math The VAE ELBO objective (Eq. 3) describes TVAE training.
- domain assumption The 1,000-row subset represents the full 19,735-row energy dataset.
- domain assumption The regression target variable is the one the evaluator intended.
- domain assumption Each generator's default hyperparameters are representative of typical library performance.
- domain assumption The custom statistical similarity score captures fidelity.
Cite this review
Pith. "Pith review of A Comparative Study of Open-Source Libraries for Synthetic Tabular Data Generation: SDV vs. SynthCity." pith.science (2026). https://pith.science/paper/3ZVQSJPD
@misc{pith2026250617847,
author = {Pith},
title = {Pith review of: A Comparative Study of Open-Source Libraries for Synthetic Tabular Data Generation: SDV vs. SynthCity},
year = {2026},
howpublished = {\url{https://pith.science/paper/3ZVQSJPD}},
note = {Machine review of arXiv:2506.17847}
}
read the original abstract
High-quality training data is critical to the performance of machine learning models, particularly Large Language Models (LLMs). However, obtaining real, high-quality data can be challenging, especially for smaller organizations and early-stage startups. Synthetic data generators provide a promising solution by replicating the statistical and structural properties of real data while preserving privacy and scalability. This study evaluates the performance of six tabular synthetic data generators from two widely used open-source libraries: SDV (Gaussian Copula, CTGAN, TVAE) and Synthicity (Bayesian Network, CTGAN, TVAE). Using a real-world dataset from the UCI Machine Learning Repository, comprising energy consumption and environmental variables from Belgium, we simulate a low-data regime by training models on only 1,000 rows. Each generator is then tasked with producing synthetic datasets under two conditions: a 1:1 (1,000 rows) and a 1:10 (10,000 rows) input-output ratio. Evaluation is conducted using two criteria: statistical similarity, measured via classical statistics and distributional metrics; and predictive utility, assessed using a "Train on Synthetic, Test on Real" approach with four regression models. While statistical similarity remained consistent across models in both scenarios, predictive utility declined notably in the 1:10 case. The Bayesian Network from Synthicity achieved the highest fidelity in both scenarios, while TVAE from SDV performed best in predictive tasks under the 1:10 setting. Although no significant performance gap was found between the two libraries, SDV stands out for its superior documentation and ease of use, making it more accessible for practitioners.
Figures
Reference graph
Works this paper leans on
-
[1]
Quality of Data in Machine Learning
Antti Kariluoto, Arto Pärnänen, Joni Kultanen, Jukka Soininen, and Pekka Abrahamsson. Quality of data in machine learning.arXiv preprint arXiv:2112.09400, 2021
work page Pith review arXiv 2021
-
[2]
Marzyeh Ghassemi, Tristan Naumann, Peter Schulam, Andrew L. Beam, Irene Y. Chen, Elias Chaibub Neto, and Mihaela van der Schaar. Opportu- nities in machine learning for healthcare.arXiv preprint arXiv:1806.00388, 2018
arXiv 2018
-
[3]
Mihaela Cătălina Stoian, Alexandru Gîrbea, and Anca D. Jurcut. A survey on tabular data generation: Utility, alignment, fidelity, privacy, and beyond. arXiv preprint arXiv:2503.05954, 2025
arXiv 2025
- [4]
-
[5]
Data Augmentation Approaches in Natural Language Processing: A Survey
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. Data augmentation approaches in natural language processing: A survey. arXiv preprint arXiv:2110.01852, 2021
work page Pith review arXiv 2021
-
[6]
OpenAI. ChatGPT. https://openai.com/index/chatgpt/, 2022. Retrieved from https://openai.com/index/chatgpt/
work page 2022
-
[7]
A. Shaji George and A. S. Hovan George. A review of chatgpt ai’s impact on several business sectors.Partners Universal International Innovation Journal, 1(1):9–23, 2023
work page 2023
-
[8]
Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, and Marah I. Abdin. On the diversity of synthetic data and its 19 impact on training large language models.arXiv preprint arXiv:2410.15226, October 2024
Show all 42 references
-
[9]
Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Je...
1901
-
[10]
Gpt-4 technical report
OpenAI. Gpt-4 technical report. Technical report, OpenAI, March 2023. Accessed: 2025-06-21
2023
-
[11]
Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Tom Henighan, Tristan Hume, David Krueger, Ben Mann, Jacob DasSarma, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022
2022 arXiv
-
[12]
Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712, 2023
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yuanzhi Lee, Yin Tat Li, Scott Lundberg, Harsha Nori, et al. Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712, 2023
2023 arXiv
-
[13]
Will we run out of data? an analysis of the limits of scaling datasets in machine learning.arXiv preprint arXiv:2211.04325, 2022
Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? an analysis of the limits of scaling datasets in machine learning.arXiv preprint arXiv:2211.04325, 2022
2022 arXiv
-
[14]
A review of the challenges with massive web-mined corpora used in large language models pre-training
Michał Perełkiewicz and Rafał Poświata. A review of the challenges with massive web-mined corpora used in large language models pre-training. arXiv preprint arXiv:2407.07630, 2024
2024 arXiv
-
[15]
Identify, align, and integrate: Match- ing knowledge graphs to commonsense reasoning tasks
Lisa Bauer and Mohit Bansal. Identify, align, and integrate: Match- ing knowledge graphs to commonsense reasoning tasks. arXiv preprint arXiv:2104.10193, 2021
2021 arXiv
-
[16]
Best practices and lessons learned on synthetic data for language models.arXiv preprint arXiv:2404.07503, 2024
Ruibo Liu, Dian Yu, Zihan Yuan, Jinchao Zhang, Fei Gao, Rui Shu, Graham Neubig, and Taylor Berg-Kirkpatrick. Best practices and lessons learned on synthetic data for language models.arXiv preprint arXiv:2404.07503, 2024
2024 arXiv
-
[17]
Machine learning for synthetic data generation: A review.arXiv preprint arXiv:2302.04062, 2024
Yingzhou Lu, Xin Meng, Yu Yao, Cong Liu, Xuefeng Li, Haifeng Yang, Zheng Wang, Bin Hu, and Qingpeng Zhang. Machine learning for synthetic data generation: A review.arXiv preprint arXiv:2302.04062, 2024. 20
2024 arXiv
-
[18]
Comprehensive exploration of synthetic data generation: A survey
André Bauer, Julian Schäfer, Michael Koch, Hanlin Yang, Samuel Kounev, Clemens Klabunde, Marco Rauh, Niko Sunderhauf, Patrick Esser, Ian Foster, and Dong Kim. Comprehensive exploration of synthetic data generation: A survey. arXiv preprint arXiv:2401.02524, 2024
2024 arXiv
-
[19]
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. InAdvances in Neural Information Processing Systems (NeurIPS), volume 27, pages 2672–2680, 2014
2014
-
[20]
Unsupervised represen- tation learning with deep convolutional generative adversarial networks
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised represen- tation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015
2015 arXiv
-
[21]
Conditional generative adversarial nets
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014
2014 arXiv
-
[22]
Wasserstein gan
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017
2017 arXiv
-
[23]
Kingma and Max Welling
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013
2013 arXiv
-
[24]
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 6840–6851, 2020
2020
-
[25]
Gomez, Łukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pages 5998–6008, 2017
2017
-
[26]
Comparison of synthetic data generation techniques for control group survival data in oncology clinical trials: Simulation study
Ippei Akiya, Takuma Ishihara, and Keiichi Yamamoto. Comparison of synthetic data generation techniques for control group survival data in oncology clinical trials: Simulation study. JMIR Medical Informatics, 12:e55118, 2024
2024
-
[27]
Copula Theory and Its Applications, volume 198
Piotr Jaworski, Fabrizio Durante, Wolfgang Karl Härdle, and Tomasz Rychlik. Copula Theory and Its Applications, volume 198. Springer, New York, 2010
2010
-
[28]
The synthetic data vault
Neha Patki, Roy Wedge, and Kalyan Veeramachaneni. The synthetic data vault. In2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pages 399–410. IEEE, 2016
2016
-
[29]
Jim Young, Sudhakar Iyengar, Mouna Mikou, and Jerome P. Reiter. Using bayesian networks to create synthetic data.Journal of Official Statistics, 25(4):549–567, 2009. 21
2009
-
[30]
Procopiuc, Divesh Srivastava, and Xiaokui Xiao
Jun Zhang, Graham Cormode, Cecilia M. Procopiuc, Divesh Srivastava, and Xiaokui Xiao. Privbayes: Private data release via bayesian networks. ACM Transactions on Database Systems (TODS), 42(4):1–41, 2017
2017
-
[31]
Modeling tabular data using conditional gan
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veera- machaneni. Modeling tabular data using conditional gan. InAdvances in Neural Information Processing Systems (NeurIPS), volume 32, Vancouver, Canada, 2019
2019
-
[32]
Ctab-gan+: Enhancing tabular data synthesis.Frontiers in Big Data, 6, 2024
Zilong Zhao, Aditya Kunar, and Robert Birke. Ctab-gan+: Enhancing tabular data synthesis.Frontiers in Big Data, 6, 2024
2024
-
[33]
Courville
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. InAdvances in Neural Information Processing Systems, volume 30, pages 5767–5777. Curran Associates, Inc., 2017
2017
-
[34]
Pacgan: The power of two samples in generative adversarial networks
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. InAdvances in Neural Information Processing Systems, volume 31, pages 14953–14963. Curran Associates, Inc., 2018
2018
-
[35]
Rigorous experi- mental analysis of tabular data generated using tvae and ctgan.International Journal of Advanced Computer Science and Applications (IJACSA), 15(4), 2024
Parul Yadav, Richa Vohra, Parul Jain, and Rajan Gupta. Rigorous experi- mental analysis of tabular data generated using tvae and ctgan.International Journal of Advanced Computer Science and Applications (IJACSA), 15(4), 2024
2024
-
[36]
Candanedo, Véronique Feldheim, and Dominique Deramaix
Luis M. Candanedo, Véronique Feldheim, and Dominique Deramaix. Data driven prediction models of energy use of appliances in a low-energy house. Energy and Buildings, 140:81–97, 2017
2017
-
[37]
Synthetic data, real errors: How (not) to publish and use synthetic data
Boris van Breugel and Mihaela van der Schaar. Synthetic data, real errors: How (not) to publish and use synthetic data. arXiv preprint arXiv:2305.09235, 2023
2023 arXiv
-
[38]
Deepenergy: Prediction of appliances energy with long-short term memory recurrent neural network
Erdenebileg Batbaatar, Hyun Woo Park, Dingkun Li, Keun Ho Ryu, and Jaeho Kim. Deepenergy: Prediction of appliances energy with long-short term memory recurrent neural network. InIntelligent Information and Database Systems, volume 10752 ofLecture Notes in Computer Science, pag...
2018
-
[39]
Bayesian network for predicting energy consumption in schools in flori- anópolis – brazil
Matheus Soares Geraldi, Mateus Vinicius Bavaresco, and Enedir Ghisi. Bayesian network for predicting energy consumption in schools in flori- anópolis – brazil. InProceedings of Building Simulation 2019: 16th Confer- ence of IBPSA, pages 4188–4195, Rome, Italy, 2019. Internatio...
2019
-
[40]
Synthcity: ctgan_plugin.py and tvae_plugin.py
SynthCity Contributors. Synthcity: ctgan_plugin.py and tvae_plugin.py. https://github.com/vanderschaarlab/synthcity, 2023. Accessed: 2025-06- 21. 22
2023
-
[41]
CTGAN and TVAE Documentation
SDV Developers. CTGAN and TVAE Documentation. https://docs.sdv.dev,
-
[2024]
Accessed: 2025-06-21. 23
2025
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.