{"id":"e003563f-53c8-456c-9b62-92fd2de07753","arxiv_id":"2505.02659","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A probability-driven prompting method, where an LLM estimates conditional categorical distributions and rows are sampled from them, outperforms table-wide and cell-by-cell generation on a California demographics dataset.","lead":"This note tests three ways of generating synthetic tabular data with an LLM and finds that asking the model for probability distributions first, then sampling rows, preserves demographic patterns better than generating whole tables or cells one by one. The method also slashes the number of API calls needed, making it attractive for large synthetic datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison is confounded by pretraining memorization: the target distribution is public Census data likely present in gpt-4o's training, and the method never conditions on the target dataset, so the reported fidelity may reflect retrieval rather than a general tabular-generation…","rationale":"The reader's weakest assumption identified the same load-bearing concern: the LLM's probability estimates must be accurate, and the method never conditions on the actual dataset. My read agrees with that assessment. The paper is transparent about the limitation, stating in Section 4 that the approach works within the bounds of the model's pre-training, but the consequence is that the central claim about 'statistically accurate tabular data generation' is only demonstrated for a public, widely known distribution. The California case cannot rule out the possibility that the method succeeds by retrieving memorized demographic statistics. I do not think this concern invalidates the demonstrated California result, which appears genuine and visually strong, but it does mean the broader claim of general statistical accuracy remains unproven. The proposed concrete test would settle whether the method can produce accurate conditional distributions when the target domain is not part of the LLM's training data. Since the reader already assigned a CONDITIONAL verdict reflecting this uncertainty, my conclusion does not change the verdict.","tokens_in":6034,"tokens_out":7000,"duration_ms":84632,"concrete_test":"Run the same probability-driven pipeline on a target distribution that is unlikely to be memorized by the LLM, for example a 2025 administrative survey released after the model's training cutoff or a synthetic contingency table with obfuscated category labels and no publicly available frequency tables. Compare the LLM's predicted conditional probabilities against the true table using total variation distance or chi-square statistic. If the error is comparable to the California case, the concern is resolved; if the error is substantially larger, the California result is likely attributable to memorized statistics rather than to the prompting method itself.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method's only source of information about the joint distribution of age and ethnicity is the LLM's parametric memory; no rows, aggregates, or examples from the target dataset are supplied during inference. Section 4 explicitly limits the approach to 'within the bounds of the model's pre-training.' The evaluation uses California age/ethnicity from the US Census Bureau, a widely reproduced public statistic that is almost certainly present in gpt-4o's training data. Thus the close match in Figure 1d and Table 1 is fully consistent with the LLM recalling known percentages rather than the probability-driven pipeline acting as a reliable general-purpose tabular data generator. The strongest claim therefore rests on an untested assumption: that the LLM's conditional probability estimates are accurate for arbitrary target domains. The demonstrated case is exactly the case where retrieval is easiest, and the absence of any non-memorized dataset or any data-conditioning step means the experiment cannot distinguish the quality of the method from the model's memorization of California demographics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 'probability-driven prompting' method for LLM-based tabular data generation. Instead of asking the LLM to emit a table all at once or to generate cells one by one, the method first prompts the LLM for marginal and conditional probability distributions over categorical features, then samples rows locally from those distributions. The method is evaluated on a simple California demographic dataset (three columns: State, Age Group, Ethnicity Group) using OpenAI's gpt-4o, with five runs of 10,000 rows per method. The reported result, based on means and standard deviations in Table 1 and the visual comparison in Figure 1, is that the probability-driven method reproduces age-dependent ethnic composition better than table-wide or cell-by-cell prompting, while using only five to six LLM calls regardless of the number of generated rows. Section 4 notes that the method works within the bounds of the model's pre-training and could be fine-tuned on domain-specific data.","tokens_in":6203,"tokens_out":4056,"duration_ms":47319,"significance":"If the claimed result holds generally, the paper describes a simple and computationally attractive way to use LLM parametric knowledge for categorical tabular data generation: the number of LLM invocations is constant in the number of output rows, and the method is transparent enough to be reproduced from the supplied prompts and pseudocode. The paper also ships reproducible scripts and generated data, which is a concrete strength. However, the significance is currently limited by the evaluation: one public dataset with three columns, one LLM, no statistical significance testing, and no comparison against the state-of-the-art methods cited in the introduction. In addition, the California age/ethnicity distribution is a well-known public statistic that is likely memorized by gpt-4o, so the reported accuracy may reflect retrieval from pretraining rather than a general capability of the prompting method. The central idea is plausible, but the evidence presented is not yet sufficient to support the abstract's general claim.","major_comments":[{"comment":"The evaluation is confounded by pretraining memorization. The target distribution is the public California age/ethnicity distribution from the Census (reference [17]), which is very likely present in gpt-4o's training data. The proposed method never conditions on the target dataset during inference: Listing 3 provides only feature names and categories, and the pseudocode in Listing 4 samples from LLM-produced probabilities without any data conditioning. The close match in Figure 1d and Table 1 is therefore fully consistent with the LLM recalling known population statistics, and it does not by itself demonstrate that the probability-driven pipeline is a generally reliable tabular data generator. Section 4 explicitly limits the method to 'within the bounds of the model's pre-training.' To support the abstract's claim of improved statistical fidelity for LLM-generated tabular data, the authors should include at least one experiment on a distribution that is not plausibly memorized, such as a synthetic distribution with known ground truth or a private dataset unavailable in pretraining.","section":"§3, Figure 1 and Table 1"},{"comment":"No statistical significance tests or quantitative error measures are reported. The paper compares methods only through means and standard deviations across five runs. The claim that the probability-driven method 'preserves age-dependent demographic heterogeneity most correctly' would be substantially strengthened by a distributional divergence measure (e.g., total variation distance or chi-square statistic) between each synthetic distribution and the Census target, with confidence intervals or paired tests across runs. Without such measures, the reader cannot determine whether the visual difference in Figure 1 is statistically credible.","section":"§3, Table 1"},{"comment":"The '−' entries in the cell-by-cell columns are ambiguous. For some rows (e.g., Children (0–17)) all six ethnicity categories are marked '−', while other rows contain a mixture of numeric values and '−' entries. The authors should state explicitly what '−' means: whether it denotes a zero count, a category that was never generated, or an omitted value. If these are zeros, they should be reported as '0.0' so that the full distribution can be inspected and compared across methods. The current presentation makes a direct comparison of the cell-by-cell method against the other two methods impossible for several age groups.","section":"§3, Table 1"},{"comment":"The method's output quality depends entirely on the LLM's probability estimates being accurate and well-calibrated in the target domain, but the paper does not assess calibration or sensitivity of those estimates. The five to six prompts are parsed into probabilities that are then used to sample all rows; if the LLM's estimates are biased, the entire synthetic dataset inherits that bias. The manuscript should report some check of the LLM's probability output quality, for example by comparing the LLM's predicted marginals against the Census marginals before sampling, and by describing the temperature/decoding settings used when eliciting the probabilities. This is directly load-bearing because no correction step or data conditioning is present.","section":"§4, Listing 4"}],"minor_comments":[{"comment":"The chain-rule formula p(w1, ..., wn) = \\prod_{k=1}^n p(w_k | w_1, ..., w_{k-1}) is rendered incorrectly in the manuscript ('nY k=1'); please fix the math typesetting.","section":"§2, Equation (1)"},{"comment":"The title 'Pseudocode for probabilistic generation of categoical tabular data' contains a typo: 'categoical' should be 'categorical'.","section":"Appendix C, Listing 4"},{"comment":"There are several spacing artifacts in acronyms such as 'V AEs' and 'V AEs are designed'; please correct to 'VAEs'.","section":"§1, Introduction"},{"comment":"The text says the reference is the US Census Bureau, but reference [17] is listed as a Public Policy Institute of California report. Please clarify the source and provide the specific Census table or citation.","section":"§3, Text after Table 1"},{"comment":"The prompt text 'reflecting population in 'State' of California/CA' is grammatically awkward; consider rewording to 'reflecting the population of California/CA'.","section":"Appendix B, Listing 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a very short empirical note, and its main limitation is the single public California Census dataset, which is likely memorized by the LLM. The stress-test concern about memorization is real and is acknowledged in Section 4. I would like to see the authors either add a non-memorized dataset or explicitly reframe the contribution as 'extracting parametric knowledge from LLMs via probability prompts' rather than general 'statistically accurate tabular data generation.' The latter framing, as currently written in the abstract, overclaims relative to the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a two-day experiment turned into a note, and I mean that as a compliment. The idea is simple and clearly stated: instead of asking an LLM to emit rows, ask it once for conditional probability tables and then sample locally. That decouples LLM cost from row count, which is a real practical win that I have not seen sharpened this way in the cited tabular-generation literature.\n\nWhat the paper does well: the comparison against table-wide and cell-by-cell prompting is the right minimal control, five runs with means and standard deviations are reported, code and data are linked, and Section 4 openly says the method works \"within the bounds of the model's pre-training.\" That is honest. The constant-query property is the strongest part of the contribution and stands on its own.\n\nSoft spots: the evaluation is too narrow for the title. One dataset, one LLM, three columns, no significance tests. The bigger issue is the memorization confound. Age-by-ethnicity for California is a widely reproduced public census table, almost certainly present in gpt-4o's training. Since the method never conditions on the target dataset and relies entirely on parametric memory, the close fit in Figure 1d is exactly what you would expect from retrieval. The paper acknowledges the pre-training limitation in words, but the demo cannot distinguish \"probability-driven prompting works\" from \"the model remembered California.\" You would need at least one dataset the model is unlikely to have memorized, or a comparison where both methods condition on the same data.\n\nAlso missing: proper baselines from the cited state-of-the-art (GReaT, Tabula, CTGAN). Comparing only to two naive prompting strategies makes the accuracy claim stronger than the evidence supports. The efficiency point, by contrast, is solid and does not depend on the accuracy comparison.\n\nWho should read this: people building LLM-based synthetic data pipelines who want a low-cost way to generate categorical data from the model's parametric knowledge. As a full answer to synthetic tabular generation, no. I would send it to peer review as a short note, with reviewers asked to focus on the memorization confound and to request an additional non-memorized dataset or a clear scoping down of the claim.","headline":"A simple, honest one-dataset note on prompting LLMs for conditional distributions; the demonstration is confounded by memorization, but the constant-query trick is worth refereeing.","tokens_in":6712,"tokens_out":2281,"would_cite":false,"duration_ms":27304,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompting an LLM for conditional probability distributions, not rows, preserves real categorical dependencies with only five or six queries.","keywords":["synthetic tabular data","large language models","probability-driven prompting","conditional probability distributions","categorical data generation","demographic statistics","LLM sampling"],"falsifier":"Re-run the experiment with a known population distribution that the model has almost certainly not memorized, such as a fictional state with a deliberately unusual age-by-ethnicity composition, and check whether the LLM's returned probabilities track that table; if the generated rows deviate systematically, the method's success in California is due to memorized census knowledge rather than to the prompting mechanism.","tokens_in":5822,"feed_emoji":"📊","tokens_out":12028,"duration_ms":118710,"temperature":0.7,"pith_summary":"This paper contends that LLM-based synthetic tabular data fails at categorical dependencies because the models are asked to generate rows auto-regressively, when their real strength lies in estimating distributions. The proposed method prompts the model for conditional probabilities—for example, $P(\\text{Ethnicity} \\mid \\text{Age Group})$—and then samples the final rows from those distributions. In a California age-by-ethnicity test, this preserves the true demographic pattern much better than table-wide prompting or cell-by-cell prompting, while requiring only five or six LLM calls no matter how many rows are produced. The practical implication is that statistically faithful synthetic data can be cheap to produce whenever the model already knows the underlying real-world distribution.","feed_headline":"LLM probability prompts match census demographics in 5-6 queries","feed_subtitle":"Synthetic demographic tables keep age-by-ethnicity structure using far fewer LLM calls than row-level generation.","key_machinery":"The central object is a conditional probability table: the LLM is prompted once per feature to output normalized probabilities, such as $P(\\text{Ethnicity} \\mid \\text{Age Group})$ for each age group, and rows are drawn from those multinomial distributions. This converts the LLM from an auto-regressive sequence sampler into a distribution estimator, which is what makes the generation cost independent of row count.","core_discovery":"The paper demonstrates that the right unit of prompting for categorical tabular data is the probability distribution, not the row. Instead of asking the model to emit a complete table or to fill cells one by one, the proposed pipeline asks a pretrained LLM to estimate marginal and conditional distributions—first the distribution of age groups, then, for each age group, the distribution of ethnicity—and then samples the rows locally from those distributions. In the California case study, this preserves the empirically observed decline in the Latino share and rise in the White share with age, while table-wide prompting over-smooths and cell-by-cell prompting flattens the table toward uniformity. The paper reports that the probability-driven method does this with five or six distributional queries, independent of the number of rows generated.","pith_inferences":["Because the method relies on the model's memorized joint distribution, an obvious unstated extension is to ground the prompts in a small real sample, correcting the recalled probabilities rather than trusting them wholesale.","The approach converts each column into a multinomial factor, so a natural stress test—one the paper does not run—is to vary the number of conditioning variables and observe when the query count grows beyond the practical limit.","If probability estimates are better calibrated than token-level samples, the method should improve most for rare categories; this is a testable prediction for future work, not something the paper claims."],"forward_implications":["The number of LLM calls stops scaling with row count: after the five or six distributional queries, any number of rows can be sampled locally, so million-row tables cost about the same as 10,000-row tables.","Categorical correlations are captured explicitly, because ethnicity is sampled from the model's estimate of $P(\\text{Ethnicity} \\mid \\text{Age Group})$, so the synthetic table retains the age-dependent composition instead of flattening it.","The method works without fine-tuning, drawing on the model's pretrained knowledge, which makes it a drop-in alternative for domains whose distributions the model has already learned.","The same pipeline can be extended to more columns by conditioning each new feature on the ones already generated, keeping the query count tied to conditioning contexts rather than to the number of rows."],"supporting_citations":[{"why":"Supplies the census-derived reference table of California's age-by-ethnicity composition that all generation methods are compared against.","marker":"[17]"},{"why":"Defines the two benchmark prompting strategies, table-wide and cell-by-cell, whose token-level failure modes the new method addresses.","marker":"[7]"},{"why":"Introduces auto-regressive LLM row generation, the representative approach whose sequential token sampling motivates moving to distribution-level prompts.","marker":"[13]"},{"why":"Documents a prompting strategy for imbalanced categorical classes, the closest existing use of prompts to steer category balance that the paper builds on.","marker":"[16]"}],"fun_headline_variants":["Probability prompts beat row-by-row for LLM tabular data","LLMs synthesize tabular data via probability distributions, not rows","Ask LLMs for distributions, not rows, to generate accurate tables","Five queries: LLM distribution prompts nail census demographic structure"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pretrained LLM's probability estimates for the target domain are accurate; the pipeline never conditions on the actual dataset, so if the model has not internalized the true joint distribution, the synthetic data will be wrong even though the sampling logic is sound.","fun_headline_variants_meta":{"raw":{"variants":["Probability prompts beat row-by-row for LLM tabular data","LLMs synthesize tabular data via probability distributions, not rows","Ask LLMs for distributions, not rows, to generate accurate tables","Five queries: LLM distribution prompts nail census demographic structure"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000752,"raw_usage":{"total_tokens":3250,"prompt_tokens":750,"completion_tokens":2500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":366,"completion_tokens_details":{"reasoning_tokens":2429}},"tokens_in":366,"tokens_out":2500,"duration_ms":20298,"temperature":1.0,"reasoning_tokens":2429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:44:26.474218+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the experiment with a known population distribution that the model has almost certainly not memorized, such as a fictional state with a deliberately unusual age-by-ethnicity composition, and check whether the LLM's returned probabilities track that table; if the generated rows deviate systematically, the method's success in California is due to memorized census knowledge rather than to the prompting mechanism.","supporting_citations":[{"cited_title":"California’s population, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the census-derived reference table of California's age-by-ethnicity composition that all generation methods are compared against."},{"cited_title":"Language models are realistic tabular data generators","cited_arxiv_id":null,"evidence_quote":"Introduces auto-regressive LLM row generation, the representative approach whose sequential token sampling motivates moving to distribution-level prompts."}],"review_version":1}