REVIEW 5 major objections 6 minor 2 cited by
FASTGEN: Fast and Cost-Effective Synthetic Tabular Data Generation with LLMs
T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A single LLM-written sampling script can replace per-record generation for synthetic tabular data, cutting cost by orders of magnitude.
desk verdict FASTGEN's script-generation trick is a real efficiency win, but the abstract oversells 'data realism' when the evaluation only measures per-field marginals and the method's own limitation section admits the joint distribution is lost. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the reusable field-level sampling script. After the LLM classifies a column as numerical, categorical, or free text, it emits Python code that encodes the inferred distribution: for numerical fields that means a parametric family with estimated parameters, for categorical fields a top-k probability table, and for free text a rule-based or library-based text generator. The script is a one-time artifact that makes the marginal cost of an extra record essentially zero and gives users a readable, editable capture of the inferred distribution.
What would settle it
Run FASTGEN on the same nine datasets with the preprocessing samples withheld (metadata descriptions only). If its KL divergence and optimal-transport distances no longer beat direct-gen, the realism advantage comes from the ground-truth samples, not from distribution inference.
Extended reading notes
Core claim
FASTGEN's central claim is that a large language model can turn a table's schema and enriched metadata into executable sampling scripts that reproduce each field's distribution, and that these scripts outperform direct record-by-record LLM generation on diversity and realism while cutting inference cost by orders of magnitude. The pipeline first prompts the LLM to act as a dataset curator and enrich terse metadata using s=100 randomly sampled ground-truth values per field. The LLM then classifies each field as numerical, categorical, or free text and writes field-specific Python code: parametric samplers (uniform, normal, Poisson) for numerical fields, top-k category samplers for categorical fields, and creative text generators that may draw on external libraries such as Faker for free text. Scripts are validated with up to n=3 automated retries, combined, and executed to produce the final dataset. Across nine datasets, FASTGEN achieves higher vocabulary, lower inter-sample n-gram redundancy, and lower KL divergence (numerical fields) and optimal-transport distance (categorical fields) than direct-gen in most cases, with Hazard Mitigation as the noted exception.
Load-bearing premise
The method's quality depends on having about 100 randomly sampled real values from every field during preprocessing; without those samples the LLM cannot infer the distributions that make the generated data realistic.
Editorial extensions
If this is right
- Generating 100,000 synthetic records would cost under $0.10 and less than an hour with FASTGEN, versus roughly $55 and 40 hours for direct per-record generation on a hosted Llama-70B, making large-scale test-data production practical.
- The approach adapts to a new domain with just a prompt and a schema; no fine-tuning or distillation is needed, so it can be applied quickly to production tables with metadata descriptions.
- High-cardinality fields such as unique IDs and free-text columns benefit most, because scripted sampling preserves full-value diversity while direct generation collapses to a small repeated vocabulary.
- The Hazard Mitigation result is a boundary case: fields with complex, non-parametric distributions can still favor direct generation, so the method's advantage is strongest for fields that fit simple distributional families.
- Because the generated scripts are exposed to users, the pipeline supports a human-in-the-loop workflow: a user can inspect, edit, or re-parameterize a script to incorporate domain knowledge that the metadata descriptions missed.
Reading between the lines
- FASTGEN is best understood as a distribution-compression method rather than a pure metadata-to-data generator: its realism claims depend on the s=100 ground-truth samples per field, and without those samples the method's advantage over direct generation would likely shrink substantially.
- The independent-field design means combinations across columns (city with state, product with category) will sometimes be implausible; capturing the most important cross-field dependencies with a two-pass dependency map is a natural next step that could close most of the remaining realism gap.
- The token-cost comparison understates the advantage at very large scale, because script generation is a fixed one-time cost; the 60x figure grows roughly linearly with record count, making FASTGEN progressively more attractive for million-row production workloads.
- A testable extension is to apply the same script-based strategy to time-series or longitudinal tabular data, checking whether autocorrelation and trend structure can be encoded in a sampling script or whether a hybrid generator is required.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FASTGEN, a method for synthetic tabular data generation that uses an LLM to (i) enrich field metadata from a small sample of ground-truth values, (ii) classify each field as numerical, categorical, or free-text, (iii) generate a Python sampling script that encodes the inferred distribution, and (iv) execute the scripts to produce records without further LLM inference. The method is compared against direct-gen, which asks the same LLM to generate each record individually, on nine real-world datasets (five from Borisov et al. and four FEMA datasets). Evaluation metrics are per-field vocabulary and ISNF for diversity, per-column KL divergence for numerical fields, and per-column optimal transport for categorical fields, using 100 generated records per dataset. The paper reports that FASTGEN uses about 6x fewer tokens at 1,000 records and 60x fewer at 10,000 records, and it claims superior diversity and realism relative to direct-gen.
Significance. The efficiency contribution is credible and practically valuable: replacing per-record LLM calls with reusable sampling scripts gives a structural token reduction that is quantified with a concrete cost/latency model (Section 5.4, Table 5). The use of multiple real datasets and explicit discussion of failure modes (e.g., Hazard Mitigation in Table 3) are strengths. However, the central realism and diversity claims are not yet established by the evidence as presented: the experiments are single-run point estimates, the realism metrics cover only per-field marginals, and the method's own limitation section concedes that cross-column dependencies are lost. If the authors add repeated runs, joint-distribution metrics, and an out-of-sample evaluation, the result would be a useful contribution to applied LLM-based data generation.
major comments (5)
- [§4.3, Tables 1–4] All comparative results are single-run point estimates on 100 records per dataset. Tables 1–4 report means or distances without standard deviations, confidence intervals, random seeds, or significance tests. Since both FASTGEN and direct-gen involve stochastic LLM sampling and stochastic script execution, the reported differences (e.g., KL 0.36 vs 0.54 for Travel Customers in Table 3) may be within run-to-run noise. The abstract's claim that FASTGEN 'outperforms traditional direct methods in both diversity and data realism' is load-bearing and requires repeated runs with variance estimates and, where possible, statistical testing.
- [§5.2, §6.3] The realism evaluation is entirely per-field marginal: KL for numerical columns (Table 3), OT for categorical columns (Table 4), and vocabulary/ISNF per field (Tables 1–2). FASTGEN generates each field independently via separate scripts (Section 3.4), and Section 6.3 explicitly concedes that this 'overlooks relationships across columns' and 'can diminish the realism of the final dataset.' No experiment measures joint distribution fidelity, such as pairwise correlations, column-pair agreement, or performance on a downstream task. Consequently, the unqualified 'data realism' claim in the abstract goes beyond what the metrics can detect; direct-gen, which generates whole records in a single call, may capture inter-field dependencies that the current evaluation cannot see. The paper should either add joint-fidelity metrics or restrict the claim to per-field marginal fidelity.
- [§3.1, §4.2] During preprocessing, the LLM is given s=100 randomly sampled ground-truth values per field to enrich metadata and infer distributions. The evaluation in Section 5.2 then compares the generated data against the same ground-truth distribution. This is an in-sample fit, not a predictive or out-of-sample test, and it means the paper does not demonstrate the advertised capability of generating realistic data from metadata descriptions alone. Please evaluate on a held-out split, or at least run an ablation without the ground-truth sample, and report how quality degrades when only metadata is available.
- [§5.1.1, Table 1] The diversity claim is not well-founded because vocabulary and ISNF measure lexical dispersion, not fidelity to the target distribution. In Table 1, FASTGEN's vocabulary exceeds the ground truth in several datasets (e.g., Sick 18.37 vs 10.30; California Housing 89.80 vs 67.60; Program Deliveries 67.43 vs 60.29). Generating more unique tokens than the real data is not automatically a quality improvement: for categorical or free-text fields, out-of-support values are likely hallucinated or implausible. The authors should pair diversity metrics with support/precision measures (e.g., fraction of generated values appearing in the ground-truth support, or semantic plausibility checks) before claiming superiority in diversity.
- [§5.2.1, Table 3] The numerical realism result is not uniformly in FASTGEN's favor: on Hazard Mitigation, KL(FASTGEN||gt)=0.76 is worse than KL(direct-gen||gt)=0.42, and the paper's own explanation is that complex distributions cannot be captured by simple formulas. This contradicts the blanket superiority claim in the abstract. The comparison should be reported per field rather than only as dataset-level averages, with a count of how many of the 222 fields each method wins, and with a discussion of the distributional conditions under which FASTGEN fails.
minor comments (6)
- [§4.3] The sentence 'its from direct-gen are highly realistic' is missing a word; please revise to 'the outputs from direct-gen are highly realistic.'
- [Table 1 caption] The caption reads 'Mean vocabulary scores from FASTGEN across fields in different datasets' but the table also reports direct-gen and gt; adjust the caption accordingly.
- [§4.2] The configuration line 'we set the number of top-k fields to k = 10' should presumably read 'top-k categories' and should clarify whether this applies to categorical fields, free-text fields, or both.
- [§5.2.1] The statement that direct-gen 'output[s] only the most frequent values' is not supported by any reported analysis; either substantiate it or rephrase.
- [§6.2] The paper would be more reproducible if the exact prompt templates, LLaMA-70B checkpoint identifier, and random seeds were provided; currently these are unspecified.
- [§5.1.2] There are several typos: 'fluctation' should be 'fluctuation' and 'This suggest' should be 'This suggests'; also, 'high volume' should be hyphenated when used as a modifier in the abstract.
Circularity Check
The realism and diversity claims reduce in part to fitting on ground-truth samples; the efficiency claim is independent.
-
fitted input called prediction
[Section 3.1 (Preprocessing); Section 4.2 (Algorithm Configuration); Section 5.2 (Realism)]
"The LLM is instructed to analyzes randomly sampled ground truth values from each field and produce a concise description tailored for data generation tasks."
FASTGEN's per-field distributions are not inferred from metadata alone: during preprocessing the LLM is given s=100 randomly sampled ground-truth values per field and asked to fold their characteristics into enriched metadata and eventually into a generation script. The realism evaluation in Section 5.2 then measures KL/OT distance between FASTGEN's output and the same ground-truth population. Thus the generative script is fitted to a ground-truth sample and scored against that ground truth, while direct-gen is denied access to those samples. The reported 'outperforms ... in data realism' is substantially an in-sample fit artifact rather than an independent out-of-sample generative prediction. The token-cost and latency comparison in Table 5 is independent of this issue and remains valid.
full rationale
The efficiency claim is self-contained and not circular: token counts are measured directly from the two generation procedures and are not derived from the ground-truth samples. The circularity is limited to the quality claims. Section 3.1 shows that the LLM's 'distribution inference' is actually an enrichment conditioned on 100 ground-truth values per field, and Section 5.2 evaluates the resulting synthetic data against that same ground truth, giving FASTGEN an informational advantage over direct-gen and making the realism/diversity comparison a partially in-sample fit. The paper's own Section 6.3 further concedes that independent per-field generation overlooks cross-column relationships, narrowing the abstract's broad 'data realism' claim; this is a scope limitation rather than a separate circular step. No self-citation chain or imported uniqueness theorem is present, and the method is not a derivational system in which outputs equal inputs by definition.
Assumptions & free parameters
free parameters (4)
- number of ground truth samples per field (s) =
100
- top-k categories for categorical fields =
10
- retry attempts for script validation =
3
- evaluation sample size per dataset =
100 records
assumptions (5)
- domain assumption Field-wise independence: each column can be generated independently without modeling dependencies.
- domain assumption Ground truth samples are representative of the full dataset distribution.
- domain assumption LLM's inferred distribution parameters and generated scripts are correct and executable.
- domain assumption The evaluation metrics (vocabulary, ISNF, KL, OT) capture 'realism' and 'diversity'.
- domain assumption Azure pricing ($0.71/M output tokens) and throughput (55 tokens/s) are representative.
Cite this review
Pith. "Pith review of FASTGEN: Fast and Cost-Effective Synthetic Tabular Data Generation with LLMs." pith.science (2026). https://pith.science/paper/FYNV7OBY
@misc{pith2026250715839,
author = {Pith},
title = {Pith review of: FASTGEN: Fast and Cost-Effective Synthetic Tabular Data Generation with LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/FYNV7OBY}},
note = {Machine review of arXiv:2507.15839}
}
read the original abstract
Synthetic data generation has emerged as an invaluable solution in scenarios where real-world data collection and usage are limited by cost and scarcity. Large language models (LLMs) have demonstrated remarkable capabilities in producing high-fidelity, domain-relevant samples across various fields. However, existing approaches that directly use LLMs to generate each record individually impose prohibitive time and cost burdens, particularly when large volumes of synthetic data are required. In this work, we propose a fast, cost-effective method for realistic tabular data synthesis that leverages LLMs to infer and encode each field's distribution into a reusable sampling script. By automatically classifying fields into numerical, categorical, or free-text types, the LLM generates distribution-based scripts that can efficiently produce diverse, realistic datasets at scale without continuous model inference. Experimental results show that our approach outperforms traditional direct methods in both diversity and data realism, substantially reducing the burden of high-volume synthetic data generation. We plan to apply this methodology to accelerate testing in production pipelines, thereby shortening development cycles and improving overall system efficiency. We believe our insights and lessons learned will aid researchers and practitioners seeking scalable, cost-effective solutions for synthetic data generation.
Figures
Forward citations
Cited by 2 Pith papers
-
Declarative Outcome-Conformant Synthesis: Exact, Closed-Form Specification Satisfaction and a Conformance Benchmark
Defines outcome-conformant synthesis as exact closed-form generation of relational data matching declared aggregates via Gamma conditional-sum sampling, introduces SpecBench for measuring conformance, and shows it is ...
-
LAB-Tab: LLM-Augmented Bayesian Network Adaptation for Few-Shot Tabular Generation
A source-trained Bayesian network, augmented with LLM-proposed edges and calibrated by PPO, generates synthetic tabular data that outperforms six baselines in six ACS distribution-shift scenarios.
Reference graph
Works this paper leans on
-
[1]
David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport.Advances in Neural Information Processing Systems, 33:21428–21439, 2020
work page 2020
-
[2]
On the opportunities and risks of foundation models
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021
arXiv 2021
-
[3]
Language models are realistic tabular data generators
Vadim Borisov, Kathrin Seler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. arXiv preprint arXiv:2210.06280, 2022
arXiv 2022
-
[4]
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020
1901
-
[5]
Evaluating large language models trained on code
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021
arXiv 2021
-
[6]
Generating multi-label discrete patient records using generative adversarial networks
Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F Stewart, and Jimeng Sun. Generating multi-label discrete patient records using generative adversarial networks. In Machine learning for health- care conference, pages 286–305. PMLR, 2017
work page 2017
- [7]
- [8]
Show all 22 references
-
[9]
Name generator, 2025
Name Generator. Name generator, 2025. Accessed: 2025-02-14
2025
-
[10]
Generative adversarial networks
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139– 144, 2020
2020
-
[11]
Generative ai for synthetic data generation: Methods, challenges and the future
Xu Guo and Yiqiang Chen. Generative ai for synthetic data generation: Methods, challenges and the future. arXiv preprint arXiv:2403.04190, 2024
2024 arXiv
-
[12]
Enhancing clinical efficiency through llm: Discharge note generation for cardiac patients
HyoJe Jung, Yunha Kim, Heejung Choi, Hyeram Seo, Minkyoung Kim, JiYe Han, Gaeun Kee, Seohyun Park, Soyoung Ko, Byeolhee Kim, et al. Enhancing clinical efficiency through llm: Discharge note generation for cardiac patients. arXiv preprint arXiv:2404.05144, 2024
2024 arXiv
-
[13]
Auto-encoding variational bayes, 2013
Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013
2013
-
[14]
Medsyn: Llm-based synthetic medical text generation framework
Gleb Kumichev, Pavel Blinov, Yulia Kuzkina, Vasily Goncharov, Galina Zubkova, Nikolai Zenovkin, Aleksei Goncharov, and Andrey Savchenko. Medsyn: Llm-based synthetic medical text generation framework. In Joint European Conference on Machine Learning and Knowledge Discovery in D...
2024
-
[15]
Transforming dental diagnostics with arti- ficial intelligence: advanced integration of chatgpt and large language models for patient care
Masoumeh Farhadi Nia, Mohsen Ahmadi, and Elyas Irankhah. Transforming dental diagnostics with arti- ficial intelligence: advanced integration of chatgpt and large language models for patient care. Frontiers in Dental Medicine, 5:1456208, 2025
2025
-
[16]
The synthetic data vault
Neha Patki, Roy Wedge, and Kalyan Veeramachaneni. The synthetic data vault. In 2016 IEEE international conference on data science and advanced analytics (DSAA) , pages 399–410. IEEE, 2016
2016
-
[17]
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 9
2019
-
[18]
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020
2020
-
[19]
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019
1910 arXiv
-
[20]
A hybrid convolutional variational autoencoder for text generation
Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. A hybrid convolutional variational autoencoder for text generation. arXiv preprint arXiv:1702.02390, 2017
2017 arXiv
-
[21]
Itergen: Iterative structured llm generation
Shubham Ugare, Rohan Gumaste, Tarun Suresh, Gagandeep Singh, and Sasa Misailovic. Itergen: Iterative structured llm generation. arXiv preprint arXiv:2410.07295, 2024
2024 arXiv
-
[22]
Large language model as attributed training data generator: A tale of diversity and bias
Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems , 36:55734–55784, 2023. 10
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.