{"id":"34a06f9f-d09a-4928-89de-bd656527f413","arxiv_id":"2506.17847","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"On one energy-consumption dataset, Synthcity's Bayesian Network had the highest statistical fidelity and SDV's TVAE the best predictive utility at 1:10 scale, with no clear library winner.","lead":"This preprint benchmarks six synthetic tabular data generators from two open-source libraries, SDV and Synthcity, on a 1,000-row energy dataset. It reports that a Bayesian Network from Synthcity best preserved statistical fidelity, while SDV's TVAE gave the best predictive utility when generating 10,000 rows, and that neither library dominates overall.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The predictive-utility ranking is unverifiable as reported: the paper never states which of the 29 columns is the regression target or how the 1,000 training rows were sampled, so Tables 5 and 6 could change under a different target or sampling scheme.","rationale":"Reader's weakest assumption is correct and is exactly the load-bearing point. The manuscript gives no target column, no sampling procedure, no seed, and no repository URL despite claiming the implementation is available. Because the central ranking is a normalized TSTR comparison, the target and sampling choices are not cosmetic details: they determine every number in Tables 5 and 6. I do not see a contradiction in the reported tables beyond the missing Synthicity-TVAE raw rows, but the reproducibility gap alone justifies a CONDITIONAL verdict. The check I propose would settle whether the concern lands: by reproducing the pipeline with the documented target and sampling, the paper's headline either survives or fails. No change to the reader's CONDITIONAL verdict is needed.","tokens_in":14234,"tokens_out":6668,"duration_ms":64720,"concrete_test":"Locate the cited GitHub repository (or, failing that, the code in the submission) and verify the data-preparation script: (1) confirm the target column used for the four regressors, expected to be 'Appliances' in the UCI energy dataset; (2) confirm the exact procedure that selected the 1,000 training rows (random seed, sampling method, and any stratification). Then re-run the 1:10 TSTR evaluation of all six generators with that target and sampling protocol using a fixed seed. If TVAE (SDV) no longer has the highest predictive-utility score in Table 6, the central claim is disproved; if the repository cannot be located or does not specify these choices, the paper needs correction or revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SDV's TVAE is the best predictive-utility model in the 1:10 setting rests on the TSTR scores in Tables 5 and 6. Those scores are normalized differences from models trained on real data, so they depend entirely on (i) which column of the 29-feature UCI energy dataset is treated as the regression response and (ii) which 1,000 rows are used as the training seed. The Methods section describes the four regression models but never names the target column, and the text says only that 'a subset of these data was used' without specifying whether the 1,000 rows are random, stratified, chronological, or fixed by a seed. If, for example, the target is 'Appliances' (the standard response in this UCI dataset) but the submitted tables instead use a different column, or if the 1,000-row sample is a contiguous winter segment rather than an i.i.d. draw, every normalized MAE/MSE/R2 score in Tables 5 and 6 would shift and the ranking 'TVAE (SDV) > Gaussian Copula (SDV) > all others' could invert. The manuscript's claim that the implementation is in an accompanying GitHub repository cannot be checked because no URL is given. The claim is therefore not falsifiable from the paper as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript compares six tabular synthetic data generators drawn from two open-source libraries: Gaussian Copula, CTGAN, and TVAE from SDV, and Bayesian Network, CTGAN, and TVAE from Synthicity. The experiments use a UCI energy-consumption dataset, train each generator on 1,000 real rows, and generate synthetic data at 1:1 (1,000 rows) and 1:10 (10,000 rows) ratios. The generators are evaluated on statistical similarity and on predictive utility via a Train on Synthetic, Test on Real (TSTR) protocol with four regression models. The reported results are that Synthicity's Bayesian Network achieves the highest statistical fidelity in both scenarios, that SDV's TVAE performs best for predictive utility in the 1:10 setting, and that neither library is an overall performance winner, with SDV preferred on documentation and usability grounds.","tokens_in":14520,"tokens_out":5989,"duration_ms":54227,"significance":"If the claims held as stated, the paper would provide useful practical guidance for practitioners choosing between SDV and Synthicity. The tabulated results show internal consistency: the predictive-utility scores in Tables 5 and 6 are plausibly derived from the normalized MAE, MSE, and R2 differences in Tables 3 and 4, and the TSTR design is methodologically appropriate. However, the manuscript omits several load-bearing experimental details: the regression target column is not identified, the sampling of the 1,000-row training subset is not described, the weights of the custom statistical-similarity metric are not given, and no repository URL is provided. As written, the headline rankings are not reproducible from the paper, so the practical guidance cannot currently be verified.","major_comments":[{"comment":"The predictive-utility evaluation never states which of the 29 columns of the UCI energy dataset is the regression target, nor how the 1,000-row training subset was sampled (random, stratified, chronological, or fixed by a seed). Tables 5 and 6 are normalized differences relative to models trained on real data, so both choices affect every entry; the claim that TVAE (SDV) is the best 1:10 model is therefore not verifiable from the text as written.","section":"Methods: Comparison Process and Methods"},{"comment":"The statistical-similarity score is defined only as 'a weighted combination' of normalized mean/median/std, Kolmogorov-Smirnov, and Wasserstein differences, with no weights or aggregation details given. Tables 1 and 2, which support the headline claim that Synthicity's Bayesian Network has the highest fidelity in both settings, are rankings of this unspecified score; the weights are free parameters that should be reported.","section":"Methods: Comparison Process and Methods"},{"comment":"The text states that 'multiple evaluation rounds were conducted using repeated holdout validation' but gives no number of rounds, train/test split ratio, seeds, or number of synthetic-generation runs; it also gives no hyperparameters or package versions, and the 'accompanying GitHub repository' is not identified by URL. The SDV-versus-Synthcity differences in Tables 3 and 4 could be within stochastic variation, so the results cannot be distinguished from noise or independently checked as reported.","section":"Methods: Comparison Process and Methods"},{"comment":"The note says that 'different amounts of real data were used for training' in the 1:1 and 1:10 experiments, but the paper does not state whether the 10,000 real training rows in the 1:10 condition are a superset of the 1,000 rows used to train the generators or an independent draw. This is essential for interpreting the comparison between models trained on 1:10 synthetic data and models trained on real data.","section":"Results: Table 4 note"}],"minor_comments":[{"comment":"The section says 'four different Data Generators' although six models are evaluated; please clarify that four architectures are instantiated by two libraries.","section":"Methods: Data Generators Description"},{"comment":"The literature review describes statistical similarity as using 'reliable statistical functions provided by SDV', while the Methods and Abstract describe a 'custom' function; specify which components come from SDV and which are custom, since using a library under comparison inside the evaluation metric is a potential fairness concern.","section":"Literature Review"},{"comment":"The phrase 'no significant distinction' and 'no significant performance gap' are used without reporting any significance test; please either provide a test or replace 'significant' with a non-statistical term such as 'substantive'.","section":"Discussion & Conclusion"},{"comment":"The title uses 'SynthCity' while the body uses 'Synthicity' inconsistently; standardize the spelling for consistency.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from a reproducibility appendix containing the regression target column, the row-sampling procedure, seeds, hyperparameters, similarity-metric weights, and a working repository URL. Without these, the empirical rankings are not independently checkable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the stress-test note is right. The paper never says which of the 29 features is the regression target for the TSTR evaluation, nor how the 1,000 training rows were chosen. That is a load-bearing omission: Tables 5 and 6 are computed from normalized differences between synthetic-trained and real-trained models, so a different target or sample could change the rankings. The standard target for this UCI dataset is 'Appliances', but the paper doesn't say. Without that, the headline claim that SDV's TVAE is best at 1:10 cannot be verified.\n\nWhat is genuinely useful: this is a direct head-to-head of six off-the-shelf generators from two popular libraries, which I haven't seen in the cited literature. The scoring is internally consistent; I re-derived the predictive utility scores from Tables 3 and 4 and they match. The conclusion that no library dominates is sensible, and the paper gives a clear, narrow practical data point.\n\nThe soft spots beyond the missing target: no code URL despite the claim, no seeds, no hyperparameters, no variance over repeated holdouts. 'Repeated holdout' is mentioned but not quantified. The 'no significant gap' statement has no statistical test behind it. Using SDV's own statistical functions to score similarity is a mild fairness issue, though I don't think it changes the story much. The literature review is also long and spends a lot of time on LLMs before getting to the point.\n\nOverall, this is a small empirical benchmark with a serious reproducibility gap. The right fix is easy: state the target, describe the sampling, give the repo and seeds, and report variance. With those, it would be a reasonable practitioner-facing note. Without them, it is not publishable as is.\n\nI would send it to review with major revisions rather than desk reject, because the question is relevant and the authors clearly did the work. But I wouldn't cite it until the missing details appear.","headline":"Useful head-to-head benchmark, but the missing target variable and sampling procedure make the headline ranking unverifiable as written.","tokens_in":15005,"tokens_out":4599,"would_cite":false,"duration_ms":45681,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a six-way comparison on 1,000 energy-data rows, Synthcity's Bayesian Network reproduced the real tables most faithfully, while SDV's TVAE led when asked to generate 10,000 rows.","keywords":["synthetic data generation","tabular data","SDV","Synthcity","Bayesian networks","CTGAN","TVAE","predictive utility"],"falsifier":"Re-run the TSTR evaluation on the same energy dataset with the regression target explicitly named and with the 1,000 training rows drawn by stratified random sampling. If SDV's TVAE no longer produces the best 1:10 predictive utility score, or if Synthcity's Bayesian Network keeps that lead, the paper's headline ranking reverses.","tokens_in":14055,"feed_emoji":"📊","tokens_out":3930,"duration_ms":33424,"temperature":0.7,"pith_summary":"The paper tries to establish, through a controlled comparison on a real energy-consumption dataset, that when the goal is statistical fidelity of synthetic tabular data, Synthcity's Bayesian Network is the strongest of six generators from two open-source libraries, and that when the goal is predictive utility at tenfold expansion, SDV's TVAE is strongest. The study matters because small organizations with limited real data need guidance on which generator to trust. The paper also argues that no library has an overall performance edge, so usability and documentation become the deciding factor.","feed_headline":"Bayesian nets beat deep generators at synthetic-data fidelity","feed_subtitle":"Six generators compared on 1,000 energy rows; SDV's TVAE leads when scaling to 10,000 rows.","key_machinery":"The machinery is the evaluation protocol rather than a single generative model. Statistical similarity is a column-wise composite score that averages normalized differences in mean, median, and standard deviation, plus the Kolmogorov–Smirnov statistic and Wasserstein distance for numeric columns and mode agreement for categorical columns. Predictive utility uses the Train-on-Synthetic, Test-on-Real (TSTR) paradigm: four regression models (XGBRegressor, Random Forest Regressor, Support Vector Regressor, and Linear Regression) are trained on each synthetic dataset and tested on real data, then scored relative to the same models trained on real data. That scoring protocol is what carries all the paper's rankings.","core_discovery":"On the paper's own terms, the central discovery is a ranking: Synthcity's Bayesian Network produces tables most statistically like the real data in both the 1:1 setting (fidelity score 96.53 out of 100) and the 1:10 setting (78.25), while SDV's TVAE produces synthetic rows with the highest predictive utility when expanding from 1,000 to 10,000 rows (score 0.31), where most other generators score near or below zero. The paper also finds that statistical similarity stays acceptably high across all generators as output scales, but predictive utility falls sharply, and that the two libraries' shared models behave differently, with SDV's CTGAN and TVAE outperforming their Synthcity counterparts in the 1:10 predictive test.","pith_inferences":["Editorial inference: the Bayesian Network's dominance likely reflects the dataset's strong inter-column dependencies (energy consumption with weather and house sensors), so the ranking may not transfer to datasets with weak or nonlinear dependencies; a useful next test is repeating the protocol on a high-cardinality categorical dataset.","Editorial inference: the TSTR scoring's sensitivity to the unstated regression target means the predictive-utility ranking is less robust than the fidelity ranking; re-running with the target column declared and stratified sampling of the 1,000 training rows is a cheap falsification.","Editorial inference: the sharp drop in predictive utility at 1:10 while fidelity stays moderate suggests the fidelity metric rewards matching marginal distributions but fails to capture joint dependencies that matter for prediction; this motivates adding a dependence-preservation metric.","Editorial inference: a practical extension would be to test whether training the generators on 2,000 or 5,000 rows changes the 1:10 ranking, since the paper's 1,000-row sample may understate deep models' behavior with more data."],"forward_implications":["A practitioner who needs statistically plausible tables from a small sample should reach for Synthcity's Bayesian Network before trying deep generators.","A practitioner who needs to produce ten times more rows than they have and wants those rows to preserve predictive value should reach for SDV's TVAE.","Generating at a 1:10 scale appears safe for distributional shape (fidelity scores 70–80) but risky for downstream modeling (most predictive-utility scores are negative), so the scale ratio is a decision variable, not a default.","Because the paper finds no overall performance winner, library choice can reasonably be driven by documentation and community support, where SDV is judged stronger."],"supporting_citations":[{"why":"Introduces the SDV package and its Gaussian copula approach, which is one of the six generators tested.","marker":"[28]"},{"why":"Introduces CTGAN and TVAE, the two deep generative architectures that both libraries implement.","marker":"[31]"},{"why":"Defines the Train-on-Synthetic, Test-on-Real paradigm that the paper uses to measure predictive utility.","marker":"[37]"},{"why":"Provides the UCI energy dataset of a low-energy house in Belgium that all generators are trained and evaluated on.","marker":"[36]"},{"why":"A prior comparison of synthetic data generation methods that contextualizes Bayesian Network performance and CTGAN sensitivity on small data.","marker":"[26]"},{"why":"States Sklar's theorem, the mathematical foundation for the Gaussian copula generator.","marker":"[27]"},{"why":"Supplies the WGAN-GP training objective that CTGAN uses for stable training.","marker":"[33]"},{"why":"Introduces PacGAN, which CTGAN uses to mitigate mode collapse.","marker":"[34]"}],"fun_headline_variants":["Bayesian net tops fidelity, TVAE tops ML utility","Synthcity BN wins stats, SDV TVAE wins ML when scaling","No library gap, but SDV's TVAE leads scaled predictive utility","For synth data: Bayesian net for fidelity, TVAE for utility","Scaling synth data: TVAE wins predictive tests, BN wins fidelity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The predictive-utility ranking rests on an unstated premise: that the regression target column among the dataset's 29 features and the way the 1,000 training rows were sampled are exactly what the evaluator assumed, but the paper never names the target or the sampling procedure.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian net tops fidelity, TVAE tops ML utility","Synthcity BN wins stats, SDV TVAE wins ML when scaling","No library gap, but SDV's TVAE leads scaled predictive utility","For synth data: Bayesian net for fidelity, TVAE for utility","Scaling synth data: TVAE wins predictive tests, BN wins fidelity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000838,"raw_usage":{"total_tokens":3686,"prompt_tokens":1012,"completion_tokens":2674,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":2581}},"tokens_in":628,"tokens_out":2674,"duration_ms":20750,"temperature":1.0,"reasoning_tokens":2581,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:59:35.241429+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the TSTR evaluation on the same energy dataset with the regression target explicitly named and with the 1,000 training rows drawn by stratified random sampling. If SDV's TVAE no longer produces the best 1:10 predictive utility score, or if Synthcity's Bayesian Network keeps that lead, the paper's headline ranking reverses.","supporting_citations":[{"cited_title":"The synthetic data vault","cited_arxiv_id":null,"evidence_quote":"Introduces the SDV package and its Gaussian copula approach, which is one of the six generators tested."},{"cited_title":"Modeling tabular data using conditional gan","cited_arxiv_id":null,"evidence_quote":"Introduces CTGAN and TVAE, the two deep generative architectures that both libraries implement."},{"cited_title":"Candanedo, Véronique Feldheim, and Dominique Deramaix","cited_arxiv_id":null,"evidence_quote":"Provides the UCI energy dataset of a low-energy house in Belgium that all generators are trained and evaluated on."},{"cited_title":"Comparison of synthetic data generation techniques for control group survival data in oncology clinical trials: Simulation study","cited_arxiv_id":null,"evidence_quote":"A prior comparison of synthetic data generation methods that contextualizes Bayesian Network performance and CTGAN sensitivity on small data."},{"cited_title":"Copula Theory and Its Applications, volume 198","cited_arxiv_id":null,"evidence_quote":"States Sklar's theorem, the mathematical foundation for the Gaussian copula generator."},{"cited_title":"Courville","cited_arxiv_id":null,"evidence_quote":"Supplies the WGAN-GP training objective that CTGAN uses for stable training."},{"cited_title":"Pacgan: The power of two samples in generative adversarial networks","cited_arxiv_id":null,"evidence_quote":"Introduces PacGAN, which CTGAN uses to mitigate mode collapse."}],"review_version":1}