{"id":"9d389279-a4fd-49ee-99fc-88a41902ba0c","arxiv_id":"2504.14368","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM-generated surrogate public data, built from schema metadata alone, can substitute for traditional public data when pretraining differentially private tabular classifiers in small-data settings.","lead":"This paper shows that AI language models can create useful 'surrogate' versions of private tabular datasets using only column names, types, and value ranges, without touching the sensitive records. These surrogate datasets can substitute for missing public data when pretraining differentially private classifiers on small datasets, helping privacy-preserving analytics in healthcare, finance, and government.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Aggregate-statistics leakage is not tested; Bordt row-level memorization tests cannot rule out LLM prior contamination from published summaries of the private EDAD/WE splits.","rationale":"The reader's weakest assumption concerned LLM memorization of the private evaluation splits, supported by post-cutoff release dates and the Bordt et al. row-level tests. My concern extends beyond row-level memorization to distributional leakage: the LLM could have seen aggregate statistics or summaries of the private surveys published before the microdata release, and the reported tests do not rule this out. This is the same general risk area but a distinct and more insidious mechanism, so agreement is partial. The concern is real but does not break the paper; it requires an additional leakage check and a scoped claim. Since the reader already issued CONDITIONAL partly for this reason, the verdict should remain CONDITIONAL. If the proposed aggregate-statistics test reveals accurate LLM knowledge of the private splits, the claim would need substantial revision; if it does not, the central result stands. The paper's own Section 8.1 flags the memorization risk, and Appendix B.1.4 provides only row-level evidence, so the residual gap is documented in the manuscript itself and weighs in the conditional assessment.","tokens_in":52574,"tokens_out":10901,"duration_ms":103543,"concrete_test":"Without providing any schema, prompt each LLM for aggregate statistics of the private splits, e.g., 'In the 2023 Spanish disability survey, what percentage of respondents have a disability degree of 75% or more?' and 'What is the median job tenure in the 2023 Workplace Equity Survey?' Compare the LLM's answers to the true private marginals and to a schema-only control (same questions but with an unrelated dataset topic). If the LLM gives accurate private statistics, distributional prior contamination is confirmed and the pretraining results must be reinterpreted. A stronger but more expensive check: regenerate surrogate data with an open-weights LLM whose training corpus provably excludes EDAD/WE and related publications, and rerun the EDAD and WE epsilon=1 pretraining experiments; if the AUC advantages persist, the central claim is robust to prior contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that schema-only LLM-generated surrogate data can replace traditional public data in DP classifier pretraining without consuming privacy budget. That claim requires the surrogate data to be genuinely independent of the private splits. The paper's non-leakage evidence (Appendix B.1.4) uses the Bordt et al. (2024) methodology, which tests verbatim record-level memorization: header tests and row-completion tests. These tests cannot detect whether an LLM has internalized aggregate statistics of the private datasets, such as marginal distributions, correlations, or survey summaries. Because EDAD 2023 and WE 2023 were conducted in 2023 and released April 2024, while model cutoffs span October 2023 to April 2024, the LLMs may plausibly have been exposed to publications, news coverage, or preliminary tables containing summaries of these surveys. If so, the generated 'surrogate' data would be statistically derived from the private distribution, violating the paper's own definition of surrogate public data as generated independently of the sensitive dataset. The paper explicitly acknowledges this risk in Section 8.1 ('there is a risk that LLM memorization may lead to overly optimistic performance estimates') but treats the row-level test as sufficient. It is not sufficient for distributional leakage. This concern does not dispute that surrogate data improve AUC; it disputes whether the improvement is attributable to schema-based priors rather than to private-information leakage. Since the whole contribution is the availability of a zero-privacy-cost public-data substitute, this is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the notion of \"surrogate public data\" for differentially private (DP) tabular analytics: datasets generated from schema-level metadata only, without access to sensitive records and without consuming privacy budget. Two LLM-based generation methods are proposed: direct CSV generation and an agent-based state machine that elicits a structural causal model (SCM) from an LLM. These are compared against simpler baselines (uniform, univariate, arbitrary Bayesian network) and against traditional public data on three datasets (ACS, EDAD, WE), three DP synthesizers (PrivBayes, GEM, AIM), and an FTTransformer classifier. The main empirical finding is that surrogate public data can substantially improve DP classifier pretraining in the low-data regime, with EDAD and WE showing large AUC advantages at small epsilon, while ACS at full size shows no pretraining benefit. The paper also reports more limited usefulness for hyperparameter tuning of DP synthesizers and a mixed story for privacy-utility tradeoff estimation.","tokens_in":52775,"tokens_out":5296,"duration_ms":52182,"significance":"If the central claim holds, the paper makes a practically valuable contribution: it relaxes the common assumption that DP tabular pipelines require access to traditional public data, and it offers a concrete, automatable way to generate domain-informed priors from public schema information. The experimental design is broad and reproducible, with public code, multiple LLMs, multiple synthesizers, multiple epsilon values, and repeated runs. The authors also take the memorization issue seriously by choosing post-cutoff private splits and applying the Bordt et al. row-level memorization tests. However, the validity of the central claim depends on a load-bearing independence assumption: the LLM-generated surrogate data must not encode information from the private splits. The paper's evidence for this assumption is incomplete, and the headline quantitative comparisons lack uncertainty quantification. These issues do not overturn the empirical phenomenon but do affect how strongly the central claim can be stated.","major_comments":[{"comment":"The paper's non-leakage evidence is insufficient for the load-bearing independence requirement. The Bordt et al. (2024) tests used in Appendix B.1.4 target verbatim record-level memorization (header tests and row completion tests); they do not test whether an LLM has internalized published aggregate statistics of the private EDAD 2023 and WE 2023 releases, such as marginal distributions, correlation tables, or descriptive summaries in codebooks and press releases. This matters because EDAD 2023 and WE 2023 were released in April 2024, overlapping with the stated cutoff of Claude 3.5 Sonnet (April 2024) and postdating the surveys themselves, so exposure to public summaries of these very releases is plausible. The paper explicitly acknowledges this risk in Section 8.1 but then treats the row-level memorization test as sufficient. It is not sufficient for distributional leakage. To support the central claim that the observed gains come from schema-based priors rather than private-informative prior contamination, the authors should add tests that compare generated surrogate data against publicly available aggregate summaries, or otherwise demonstrate that the LLM's output is insensitive to the private distribution's published summary statistics.","section":"8.1, Appendix B.1.4"},{"comment":"The headline pretraining results are presented as point estimates without standard errors, confidence intervals, or significance tests, although 10 runs per configuration are reported. For example, on EDAD at epsilon=1 the AUC advantage is 0.19 for traditional public data, 0.17 for CSV (Claude), and 0.17 for CSV (Llama); the differences among these methods are small relative to the expected variance of DP training. Similarly, the WE results at epsilon=1 show Agent (Claude, Unif.) at 0.21 versus Public at 0.11, but no measure of variability is given. The claim that surrogate methods are \"on par with\" traditional public data, and the ranking of methods in Figures 3-9, would be substantially strengthened by reporting standard errors or confidence intervals and, where appropriate, paired significance tests across the shared fine-tuning hyperparameter configurations.","section":"6.1, Table 3"},{"comment":"The explanation that the ACS failure is due to dataset size rests on subsampling a single dataset (ACS) at four rates. This is suggestive but not conclusive: the observed difference between full-size ACS and the small EDAD/WE datasets could also be explained by domain, schema complexity, or the strength of the LLM's prior for the specific variables. Since the low-data regime is the core of the paper's central claim, the evidence base of two small datasets is thin. The authors should either add at least one additional large dataset (or an additional small one from a different domain) to test the size hypothesis, or explicitly scope the claim to the two demonstrated datasets and temper the abstract accordingly.","section":"6.1, Figure 9"}],"minor_comments":[{"comment":"The section title \"statical distance metrics\" contains a typo and should read \"statistical distance metrics.\"","section":"Section 3 heading"},{"comment":"The Univariate baseline is explicitly disclosed as not a valid public data surrogate, but this disclosure appears only in a footnote. Because the method appears as a Pareto efficient method in Task 3 results (e.g., Table 14), the main text should clearly mark it as a reference condition and not as a candidate surrogate.","section":"4.1.2, footnote 2"},{"comment":"In Table 16, zero values are omitted for readability, but this makes it difficult to distinguish true zero similarity from missing entries, particularly for EDAD and WE where many LLM-based rows contain only blanks. A placeholder such as \"<0.05\" or explicit zeros would improve interpretability.","section":"Table 16, Appendix D"},{"comment":"The table contains a typo: \"Agent (Allm Unif.)\" should be \"Agent (All, Unif.).\"","section":"Appendix C.1, Table 22a"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is substantial and the paper is likely to be of interest to the DP and tabular-data communities. The main risk is the leakage/contamination concern: the central claim requires the surrogate data to be generated independently of the private splits, and the current memorization tests do not rule out distributional prior contamination from published summaries of EDAD 2023 and WE 2023. I would encourage the editor to require additional contamination analysis before acceptance. The lack of uncertainty quantification in the headline tables is also important, though more readily fixable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper gives the tabular-DP community a genuinely new option. The idea of surrogate public data – synthetic data built only from schema-level metadata, consuming no privacy budget – is clearly defined, and the proposed Agent state machine that elicits a structural causal model from an LLM is a nice piece of engineering. The evaluation is unusually broad: three datasets, three LLMs, three synthesizers, five epsilons, ten runs per configuration, and the code is public. That alone justifies a serious look.\n\nThe main result, that surrogate public data can replace traditional public data for pretraining DP classifiers, is supported on the two smaller datasets (EDAD, WE). The AUC advantages are large enough that sampling noise is unlikely to explain them, especially at eps=1. I also credit the authors for choosing private splits released after the LLM cutoffs and for running the Bordt et al. memorization tests. They are open about the residual risk in Section 8.1.\n\nThe soft spots: the headline tables report means without error bars or significance tests, so some of the differences between methods are hard to interpret. The larger conceptual issue is the one the stress-test note raises. The Bordt tests only detect record-level verbatim memorization, not distributional leakage. If the LLM had seen summary tables, news coverage, or preliminary results from the 2023 EDAD or WE surveys, the generated surrogate data could be statistically derived from the private distribution, which would break the paper's own definition and inflate the pretraining gains. The authors acknowledge this risk but treat the row-level test as sufficient. That is not quite enough; a direct comparison of the generated marginals and correlations to the private splits, or an experiment with a dataset the LLM provably cannot know, would strengthen the claim. Still, this is an addressable concern, not a fundamental flaw. The failure of all methods on the full ACS dataset is honestly reported and plausibly attributed to the large dataset size.\n\nWho is this for? Anyone working on differential privacy for tabular data, especially those deploying DP classifiers in the low-data regime. The paper deserves peer review; a good referee should push for uncertainty quantification and a more convincing independence check. My bottom line: it is a solid, useful contribution, worth citing and worth engaging with.","headline":"A broad and genuinely useful empirical study of LLM-generated surrogate public data for DP tabular ML; the main pretraining result holds on small datasets, but the independence-from-private-data assumption needs stronger support.","tokens_in":53387,"tokens_out":2974,"would_cite":true,"duration_ms":26665,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated surrogate public data—synthesized from schema metadata alone—can effectively replace traditional public data when pretraining differentially private tabular classifiers, with the strongest gains on small datasets.","keywords":["differential privacy","surrogate public data","tabular data","large language models","DP pretraining","structural causal models","hyperparameter tuning","privacy-utility tradeoff"],"falsifier":"Take a private split that was published before the generation models' training cutoffs and run the paper's own header-generation and row-completion memorization tests on the LLMs used to build surrogates; if the LLMs reproduce exact or near-exact private records, the reported pretraining gains cannot be attributed to schema-based priors.","tokens_in":52321,"feed_emoji":"📊","tokens_out":8906,"duration_ms":76073,"temperature":0.7,"pith_summary":"Differentially private tabular learning usually assumes a public dataset exists for pretraining, tuning, and estimating the privacy-utility trade-off, but such public samples are often unavailable for tabular data. This paper introduces surrogate public data: datasets generated purely from schema-level metadata, consuming no privacy budget, and proposes automating that generation with large language models. The central claim is that these surrogates can effectively replace traditional public data when pretraining DP tabular classifiers, with the strongest evidence in the low-data regime: at $\\varepsilon=1$ the traditional public split on EDAD gives an AUC advantage of 0.19 while Claude and Llama CSV surrogates give 0.17, and on WE a Claude Agent surrogate gives 0.21 versus 0.11 for the public split. Surrogates also help, to a lesser extent, for hyperparameter tuning of private synthetic-data generators and for estimating the privacy-utility trade-off.","feed_headline":"LLM-made surrogate data rivals real public data for private tabular ML","feed_subtitle":"Schema-only LLM surrogates match traditional public splits when pretraining DP classifiers on small tabular datasets.","key_machinery":"The central object is surrogate public data plus a schema-to-data generation pipeline. The Agent method is the load-bearing piece: an automated state machine that prompts an LLM to declare variables, propose consistency constraints, choose root nodes, add parent-child edges while checking acyclicity, assign structural equations and parameters, emit sampling code, and enforce value ranges, re-querying on any validation failure. This encodes plausible dependencies among variables using only schema metadata, making the sampled records statistically coherent while consuming zero privacy budget. Direct CSV generation is the simpler alternative, and the uniform, univariate, and arbitrary Bayesian-network baselines isolate which data properties actually matter.","core_discovery":"On its own terms, the paper claims that LLMs can stand in for the panel of domain experts that would manually encode a plausible data-generating process, turning public schema or codebook metadata into usable public data. It proposes two automated routes: direct CSV record generation, and a state-machine Agent that elicits a structural causal model (a DAG with structural equations) from the LLM and then samples from it. Across three datasets, the paper finds that LLM-generated surrogates match traditional public data when pretraining a DP FTTransformer under DP-SGD fine-tuning at $\\varepsilon=1$ on small datasets, with the strongest aggregate results for direct CSV generation from Claude on EDAD and for the Claude Agent on the Workplace Equity survey. On the large ACS dataset, no pretraining method helps, and a subsampling analysis shows the advantage reappears at 5% and 10% sizes, which the paper attributes to the inverse relationship between noise and dataset size.","pith_inferences":["Editorial inference: If schema priors are what carry the gains, then the binding constraint becomes metadata quality, and augmenting sparse codebooks with retrieval from domain literature or expert review of the elicited SCM should push surrogates closer to parity with real public splits.","Editorial inference: The large-dataset failure implies a measurable size threshold—somewhere between the 1,400-record WE split and the 23,006-record ACS split at $\\varepsilon=1$—where surrogate pretraining stops beating no pretraining; locating that threshold across datasets would give practitioners a rule of thumb.","Editorial inference: The absence of a link between TVD and 3-way marginal similarity and usefulness suggests the right similarity measure for surrogate selection is task-relative, such as fidelity of the conditional distribution of the target given features, and such a metric could select among generated surrogates without spending privacy budget.","Editorial inference: Because CSV and Agent surrogates succeed in different spots (CSV on EDAD pretraining, Agent on WE pretraining and tuning), a portfolio that generates both and picks by task may beat either alone; the paper does not test this."],"forward_implications":["Practitioners with a small private tabular dataset and no public split can pretrain a DP classifier on LLM-generated surrogate data and keep most of the pretraining benefit that a real public sample would provide.","The input that matters is schema metadata, so a codebook or README—rather than an actual public sample—can be sufficient to build a zero-privacy-cost prior for private learning.","Surrogate pretraining is a low-data-regime tool: as the private dataset grows, the advantage disappears, so large-scale deployments should not expect gains from this technique.","For tuning DP synthesizers, encoding any plausible dependency structure can be enough to choose useful hyperparameters, which explains why an arbitrary Bayesian network lands on the Pareto frontier alongside the LLM-generated surrogates.","Estimating the privacy-utility trade-off is the hardest of the three tasks; surrogate data tracks the private curve only loosely, and traditional public data remains the best estimator in most settings."],"supporting_citations":[{"why":"Shows that LLMs can act as realistic tabular data generators, grounding the direct CSV generation method.","marker":"Borisov et al. (2023)"},{"why":"Introduces LLM-specified structural causal models that generate records from a topological order alone, the template the Agent method adapts.","marker":"Bynum & Cho (2024)"},{"why":"Provides the verbatim-memorization tests used to argue the private EDAD and WE splits were not memorized by the generation LLMs.","marker":"Bordt et al. (2024)"},{"why":"Explains why public pretraining helps private training by locating a good basin first, the theoretical motivation for the pretraining task.","marker":"Ganesh et al. (2023)"},{"why":"Supplies evidence that public pretraining beats fully private training under distribution shift, the comparison the pretraining task is built on.","marker":"Thaker et al. (2024)"},{"why":"The birth-registry deployment that motivates the need for zero-privacy-budget data for hyperparameter tuning and trade-off estimation.","marker":"Hod & Canetti (2025)"},{"why":"The practical guide whose recommendation to tune on a proxy public dataset frames the DP auxiliary task definition.","marker":"Ponomareva et al. (2023)"},{"why":"The closest contemporary study of LLM-generated public data for private synthetic data, whose negative results the paper reinterprets through dataset size.","marker":"Swanberg et al. (2025)"}],"fun_headline_variants":["LLM surrogate data: no public data needed for private tabular ML","Schema-only LLM data matches public data for private tabular models","Surrogate public data from LLMs replaces real public data in DP training","LLM-generated surrogates rival public data for private tabular learning","No public data? LLM surrogates work for DP tabular classifiers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the large language models used to generate the surrogate data never memorized the private evaluation records, so the measured gains reflect the schema-based prior rather than data leakage.","fun_headline_variants_meta":{"raw":{"variants":["LLM surrogate data: no public data needed for private tabular ML","Schema-only LLM data matches public data for private tabular models","Surrogate public data from LLMs replaces real public data in DP training","LLM-generated surrogates rival public data for private tabular learning","No public data? LLM surrogates work for DP tabular classifiers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1595,"prompt_tokens":998,"completion_tokens":597,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":501}},"tokens_in":614,"tokens_out":597,"duration_ms":4996,"temperature":1.0,"reasoning_tokens":501,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:49:49.772174+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a private split that was published before the generation models' training cutoffs and run the paper's own header-generation and row-completion memorization tests on the LLMs used to build surrogates; if the LLMs reproduce exact or near-exact private records, the reported pretraining gains cannot be attributed to schema-based priors.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces LLM-specified structural causal models that generate records from a topological order alone, the template the Agent method adapts."},{"cited_title":"Z., and Smith, V","cited_arxiv_id":null,"evidence_quote":"Supplies evidence that public pretraining beats fully private training under distribution shift, the comparison the pretraining task is built on."},{"cited_title":"and Canetti, R","cited_arxiv_id":null,"evidence_quote":"The birth-registry deployment that motivates the need for zero-privacy-budget data for hyperparameter tuning and trade-off estimation."},{"cited_title":"B., Vassilvitskii, S., Chien, S., and Thakurta, A","cited_arxiv_id":null,"evidence_quote":"The practical guide whose recommendation to tune on a proxy public dataset frames the DP auxiliary task definition."}],"review_version":1}