{"id":"5a2958b2-c8bf-4c2e-89af-545337fc59cd","arxiv_id":"2412.02467","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Warming up on pseudo-data before DP fine-tuning improves LLM-based private tabular generation on several datasets, with the out-of-distribution variant remaining DP-compliant.","lead":"DP-2Stage first fine-tunes a language model on a fake or public pseudo-table, then fine-tunes on the real table with differential privacy. It reports utility gains over direct DP fine-tuning on some datasets, but gains are inconsistent and the uniform-pseudo-data variant uses private column statistics.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DP-2Stage-U is not (ε,δ)-DP as presented because stage-1 uses private column statistics without protection; the paper's headline claim therefore holds only for DP-2Stage-O, whose utility gains over DP-Standard are inconsistent.","rationale":"I read the paper's central claim as: the two-stage fine-tuning framework DP-2Stage improves DP tabular generation by using non-private pseudo data in stage 1 to preserve structure, then DP fine-tuning on private values. The validity of this claim requires that both variants—DP-2Stage-O and DP-2Stage-U—are actually differentially private, since the abstract and Table 3 present both without qualification. The manuscript itself, in Section 6, concedes that DP-2Stage-U does not satisfy the stated DP guarantee because stage-1 statistics are computed from private data. This is a fundamental gap: the method cannot be advertised as 'differentially private' when a core component releases private information. The reader's weakest assumption identifies exactly this. I agree that the privacy guarantee is the most load-bearing concern. I also note that even setting the privacy issue aside, the empirical support is not robust: DP-2Stage-O underperforms DP-Standard on Airline, and DP-2Stage-U underperforms on Adult and Texas utility metrics. This means the phrase 'improves performance across various settings and metrics' is an overgeneralization. However, the privacy gap is more fundamental because it undermines the central contribution's defining property. A concrete check would be to make stage-1 statistics DP and re-run the benchmarks under a composed budget; if the utility gains vanish, the method's advantage is attributable to the non-private leakage, not to the two-stage DP training. Since the reader already flagged this and issued CONDITIONAL, my stress-test does not change the verdict; it strengthens the need for the authors to either privatize the statistics or explicitly scope all claims to DP-2Stage-O.","tokens_in":25545,"tokens_out":6634,"duration_ms":65215,"concrete_test":"Run DP-2Stage-U exactly as in Table 3 but compute the stage-1 column ranges and category lists with a differentially private mechanism (e.g., Laplace noise for min/max with sensitivity based on global domain size, and the exponential mechanism for categories), allocating ε=0.1 of the total budget; then run stage 2 with the remaining ε=0.9 using DP-SGD and account for the full composition under RDP. If the resulting Hist and F1 scores no longer beat DP-Standard at ε=1, or if the composed privacy accountant reports ε>1 for the same δ, then the reported gains for DP-2Stage-U are enabled by non-private leakage rather than by the two-stage DP mechanism, and the central claim must be restricted to DP-2Stage-O.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that DP-2Stage improves utility and fidelity over direct DP fine-tuning while providing differential privacy. For DP-2Stage-U (Section 4.1), the pseudo dataset is built from ranges and category lists of the private dataset, and these statistics are used non-privately. Section 6 explicitly states that this 'weakens the stated privacy guarantees' and that the statistics should be computed in a differentially private manner. Therefore, the (ε,δ)-DP guarantee, as defined in Definition 3.2 and accounted in Appendix A.2, does not hold end-to-end for DP-2Stage-U as presented. Since the abstract and Table 3 present DP-2Stage-U without this caveat, the central claim overstates the method's privacy. Even for DP-2Stage-O, which is DP, the utility improvements are mixed: on Airline, DP-2Stage-O (+adult, +texas) has lower F1, AUC, and ACC than DP-Standard (Table 3), and on Adult DP-2Stage-U underperforms DP-Standard on all utility metrics. Thus, the claim of 'improves performance across various settings' is not consistently supported. The load-bearing issue is the privacy gap of DP-2Stage-U because it invalidates a core advertised property of the proposed framework, not merely its empirical edge.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DP-2Stage, a two-stage fine-tuning framework for LLM-based differentially private tabular data generation. The first stage non-privately fine-tunes a pre-trained LLM on a pseudo dataset, constructed either by uniform sampling from private column statistics (DP-2Stage-U) or from an out-of-distribution public dataset (DP-2Stage-O). The second stage applies DP-SGD on the private dataset. The authors report improved F1, histogram intersection, and perplexity compared with direct DP fine-tuning (DP-Standard) on Adult, Airline, and Texas datasets, and they release code and experimental setup.","tokens_in":25823,"tokens_out":3123,"duration_ms":30766,"significance":"If the central claim held, the two-stage approach would offer a practical way to redirect the DP budget from table structure toward private values, which is a plausible and useful idea for LLM-based tabular synthesis. The paper is empirically grounded: it reports five runs with standard deviations, compares against several baselines, and includes a limitations section that openly discusses open problems. However, the privacy guarantee for DP-2Stage-U is not end-to-end as presented, and the utility improvements over DP-Standard are inconsistent across datasets and variants. These issues weaken the paper's headline contribution as currently stated, although the underlying framework remains of interest if the claims are appropriately scoped.","major_comments":[{"comment":"DP-2Stage-U as presented is not (epsilon, delta)-differentially private. In Section 4.1, the uniform pseudo dataset is constructed using the range of numerical columns and the category lists of the private dataset, and this construction is not protected by any DP mechanism. Section 6 explicitly states that this 'weakens the stated privacy guarantees' and recommends that the statistics be computed in a differentially private manner. Therefore, the end-to-end (epsilon, delta)-DP guarantee of Definition 3.2 does not hold for DP-2Stage-U as evaluated in Table 3. The abstract and the contributions list DP-2Stage-U as part of the proposed framework without this caveat, which overstates the privacy property. The paper should either compute the pseudo-data statistics with a dedicated privacy budget, or clearly present DP-2Stage-U as a non-DP variant and restrict the privacy claim to DP-2Stage-O.","section":"Section 4.1 and Section 6"},{"comment":"The claim that the approach 'improves performance across various settings and metrics' is not consistently supported by the reported results. In Table 3, on Adult, DP-2Stage-U is worse than DP-Standard on F1 (21.2 vs 27.8), AUC (48.9 vs 58.5), and ACC (61.9 vs 65.2). On Airline, DP-2Stage-O (+adult) is worse than DP-Standard on F1 (55.2 vs 60.5), AUC (62.5 vs 65.3), and ACC (60.0 vs 62.4). On Texas, DP-2Stage-U is substantially worse than DP-Standard on F1 (23.5 vs 55.4). The gains are clear only for DP-2Stage-O on Texas and partially on Adult, and for DP-2Stage-U on Airline. The paper should either revise the central claim to specify the settings in which improvements occur, or provide an analysis of the conditions under which the two-stage approach helps or hurts.","section":"Abstract and Table 3"}],"minor_comments":[{"comment":"There is a grammatical error: 'Our findings shows' should be 'Our findings show.'","section":"Abstract"},{"comment":"The text says 'the size of the synthethic data is the same' and later the same spelling appears; 'synthethic' should be 'synthetic.'","section":"Section 5.1"},{"comment":"In Table 4, DP-Standard and DP-2Stage-O with column shuffling are reported from a single model run, while other configurations use five runs. The absence of standard deviations for those entries is acknowledged in Section 5.4.1, but the visual formatting makes it easy to misread all entries as comparable. Adding a note or marker directly in the table would improve clarity.","section":"Table 4"},{"comment":"The discussion of lambda weighting states that DP-2Stage-U generally performs better with the default loss on Adult, but the table shows mixed results; a more precise summary of the pattern would help the reader.","section":"Section 5.4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an empirical study with a clear presentation, and the authors are transparent about the privacy caveat for DP-2Stage-U in the limitations section. However, the abstract and contributions do not carry that caveat, and several empirical results contradict the stated headline improvement. These are fixable with a revision that scopes the claims and, ideally, makes DP-2Stage-U actually DP-compliant or explicitly labels it as non-private."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read DP-2Stage. Bottom line: the two-stage idea is not new (Tran and Xiong already have it), but this paper does a careful empirical job on pseudo-data choice, column shuffling, and loss weighting under DP. The central claim, however, is overstated, and the U variant as presented is not actually DP.\n\nWhat is genuinely useful here: the comparison of uniform pseudo-data (U) against out-of-distribution pseudo-data (O), the finding that disabling column shuffling helps under DP while hurting in the non-DP setting, and the lambda study. The paper is honest in its Limitations section, explicitly saying that DP-2Stage-U uses private statistics in stage one and therefore weakens the stated guarantees, and that those statistics should be computed privately. That is the right thing to acknowledge, but it means the abstract and Table 3 present the U variant without the necessary caveat. The stress-test note is correct on this point: as it stands, DP-2Stage-U is not an (epsilon,delta)-DP mechanism.\n\nThe empirical support for the headline is also mixed. DP-2Stage-O beats DP-Standard on Adult and Texas F1 by a wide margin, but underperforms on Airline F1 (55 vs 60.5). DP-2Stage-U underperforms DP-Standard on Adult and Texas F1. So \"improves performance across various settings\" is not consistently true. The paper's own hypotheses about when each variant helps are plausible but post hoc. Given the large standard deviations, several of these differences are within noise, though the Texas O gains look real.\n\nWhat is new is the analysis, not the architecture. The core two-stage framework is already in Tran and Xiong; this paper contributes the pseudo-data comparison and the shuffling result. That is a modest but real increment, and the code release helps reproducibility.\n\nWho is this for? Researchers working on DP fine-tuning of LLMs for tabular generation. The lesson that structural tokens waste the privacy budget and that pseudo-data can redirect it is worth knowing, even if the reported gains are inconsistent. The privacy gap in the U variant is a genuine flaw, but it is openly disclosed and fixable by allocating a small budget to stage-one statistics.\n\nI would send this to a serious referee. It is a real empirical contribution with reproducible code and an honest limitations section, but the current framing overclaims. The referee should push for a rewrite that separates the DP-valid O variant from the U variant and tempers the abstract. It also deserves a careful look at the metric variance before any acceptance.\n\nMy own verdict: conditional. The core mechanism is sound, but the advertised privacy-utility trade-off holds only for one variant and is not consistent across datasets.","headline":"Solid empirical analysis of two-stage DP fine-tuning, but the headline claim only holds for the O variant and the U variant is not DP as presented.","tokens_in":26403,"tokens_out":1758,"would_cite":false,"duration_ms":20731,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two-stage tuning — learn the table template from pseudo data first, then fine-tune under DP on the private data — improves the utility and fidelity of privately generated tabular data over direct DP fine-tuning.","keywords":["differential privacy","synthetic tabular data","large language models","DP-SGD","two-stage fine-tuning","pseudo dataset","tabular data generation"],"falsifier":"Rerun DP-2Stage-U's stage one with its column-range and category-list statistics replaced by differentially private versions of the same statistics and recompute the reported $\\varepsilon$: if the effective privacy loss at $\\delta = 10^{-5}$ exceeds the claimed $\\varepsilon = 1$, the uniform variant as presented fails its own privacy claim. Separately, to test the structural-transfer story, feed DP-2Stage-O a stage-one corpus of randomly ordered non-tabular text with the same serialization template; if the F1 and Hist gains over DP-Standard vanish, the benefit is the template-learning mechanism and not merely extra fine-tuning steps.","tokens_in":25308,"feed_emoji":"🔒","tokens_out":11096,"duration_ms":97673,"temperature":0.7,"pith_summary":"Directly fine-tuning a language model under differential privacy to generate tabular data spends much of the privacy budget on tokens that are not private — column names and filler words such as 'is' and ',' — which is why the output loses coherence. The paper proposes DP-2Stage, a two-stage fine-tuning schedule in which the model first learns the table's serialization template from a non-private pseudo dataset, and then is fine-tuned on the real private data with DP-SGD while a weighted loss emphasizes value tokens. On the Adult, Airline, and Texas datasets at $\\varepsilon = 1$, the two-stage models improve F1 by 12–25% and marginal-distribution fidelity by 1–3% over directly fine-tuned DP models, and the uniform-pseudo variant generates a synthetic dataset up to 21× faster. The authors also state that only the out-of-distribution pseudo-data variant carries the full privacy guarantee as published, because the uniform variant builds its pseudo data from private column statistics.","feed_headline":"Two-stage tuning beats direct DP fine-tuning for LLM tables","feed_subtitle":"Pre-training on pseudo data spares the privacy noise for private values, lifting F1 by 12–25%.","key_machinery":"The load-bearing object is the pseudo dataset: a non-private table that instantiates the same '<key> is <value>,' serialization template as the private data, so the model can absorb the template in stage one without consuming the stage-two privacy budget (the paper flags that the uniform variant's statistics still touch private data). The second load-bearing mechanism is the stage-two weighted loss, Equation (6), which places weight $\\lambda = 0.65$ on value-token log-likelihoods and weight $1-\\lambda$ on key and filler tokens, steering the DP-SGD noise toward private content. Together these convert the privacy budget from a tax on the whole sentence into a tax on values only.","core_discovery":"The paper's central claim is that LLM-based tabular synthesis under DP fails not because the model cannot learn private values but because DP-SGD's noise is spilled on learning the table's structure — keys, 'is' tokens, commas — which are not private. DP-2Stage therefore decouples the two tasks: stage one fine-tunes a pre-trained GPT-2 on a pseudo dataset, built either by sampling uniformly from private column ranges and category lists (DP-2Stage-U) or by taking an unrelated public dataset (DP-2Stage-O), so that template learning consumes none of the stage-two privacy budget; stage two fine-tunes the result with DP-SGD on the private data using a loss weighted with $\\lambda = 0.65$ toward value tokens. Across three datasets at $\\varepsilon = 1$, $\\delta = 10^{-5}$, both variants match or exceed direct DP fine-tuning on utility and fidelity, with DP-2Stage-O best on Adult and Texas, DP-2Stage-U best among LLMs on Airline and fastest at sampling, and the paper explicitly conceding that DP-2Stage-U as presented is not fully private because its stage-one statistics come from the private data.","pith_inferences":["Read literally, DP-2Stage-U's results are what a fully private pipeline could hope to achieve only after its stage-one statistics are themselves made private; a version that computes column ranges and category lists under DP is the natural next test, and the paper does not run it.","The structure-then-content split should transfer beyond tables: DP fine-tuning of generative models over any rigidly structured output — JSON, code, schemas, forms — plausibly wastes noise on syntax, so a pseudo-data warm-up is a cheap experiment in those settings.","The finding that column shuffling hurts DP but helps non-DP training suggests order noise and gradient noise interact super-linearly, and understanding that interaction could yield DP-aware data-loading rules rather than simply disabling shuffling."],"forward_implications":["DP fine-tuning of LLMs for tabular data should spend its budget on values, not on template tokens, and the two-stage schedule is a way to do that.","Using an out-of-distribution public dataset as pseudo data keeps the method strictly DP as stated and still competitive, outperforming direct DP fine-tuning on Adult and Texas.","Using uniform pseudo data built from private column statistics makes generation up to 21× faster, which is the practical advantage of the variant if its privacy gap is closed.","Disabling column shuffling helps every DP variant at ε = 1, the opposite of the non-DP convention.","Raising the budget to ε = 8 improves DP-2Stage utility, consistent with the noise-allocation diagnosis."],"supporting_citations":[{"why":"Supplies the 'key is value,' serialization, the non-private GPT-2 baseline, and the column-shuffling scheme that DP-2Stage modifies.","marker":"Borisov et al., 2023"},{"why":"Supplies DP-SGD, the private optimizer used in stage two and in the DP-Standard and DP baseline comparisons.","marker":"Abadi et al., 2016"},{"why":"Concurrent two-stage DP LLM tabular-synthesis work that motivates the weighted value-token loss in stage two.","marker":"Tran & Xiong, 2024"},{"why":"Supplies the definition of (ε,δ)-differential privacy and the Gaussian mechanism that ground the privacy model.","marker":"Dwork et al., 2014"},{"why":"Supplies the Rényi-DP accountant used to convert DP-SGD updates into the reported (ε,δ) budgets.","marker":"Mironov et al., 2019"},{"why":"Provides the CTGAN and TVAE baselines and the ML-efficacy, Hist, Pair, and CorAcc evaluation metrics.","marker":"Xu et al., 2019"},{"why":"Provides the preprocessed Texas Hospital Discharge dataset used as one of the three evaluation datasets.","marker":"Stadler et al., 2022"},{"why":"Supplies AIM, the marginal-based DP baseline that the appendix compares the LLM methods against.","marker":"McKenna et al., 2022"}],"fun_headline_variants":["DP-2Stage: Two-step tuning improves private LLM tables","Private tables: LLMs should learn structure first","Pseudo-data stage spares DP budget for private values","Direct DP fine-tuning? LLMs prefer a two-stage approach","Split public and private: DP-2Stage for LLM tables"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For the out-of-distribution variant, the load-bearing premise is that structure learned from an unrelated public table transfers to the private table; for the uniform variant, the load-bearing premise is that the private column ranges and category lists used to build the pseudo data leak nothing — a premise the authors themselves say is false, so only the out-of-distribution variant carries the stated $(\\varepsilon,\\delta)$ guarantee as published.","fun_headline_variants_meta":{"raw":{"variants":["DP-2Stage: Two-step tuning improves private LLM tables","Private tables: LLMs should learn structure first","Pseudo-data stage spares DP budget for private values","Direct DP fine-tuning? LLMs prefer a two-stage approach","Split public and private: DP-2Stage for LLM tables"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"total_tokens":1822,"prompt_tokens":1003,"completion_tokens":819,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":735}},"tokens_in":619,"tokens_out":819,"duration_ms":8774,"temperature":1.0,"reasoning_tokens":735,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:25:01.897598+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun DP-2Stage-U's stage one with its column-range and category-list statistics replaced by differentially private versions of the same statistics and recompute the reported $\\varepsilon$: if the effective privacy loss at $\\delta = 10^{-5}$ exceeds the claimed $\\varepsilon = 1$, the uniform variant as presented fails its own privacy claim. Separately, to test the structural-transfer story, feed DP-2Stage-O a stage-one corpus of randomly ordered non-tabular text with the same serialization template; if the F1 and Hist gains over DP-Standard vanish, the benefit is the template-learning mechanism and not merely extra fine-tuning steps.","supporting_citations":[{"cited_title":"Language models are realistic tabular data generators","cited_arxiv_id":null,"evidence_quote":"Supplies the 'key is value,' serialization, the non-private GPT-2 baseline, and the column-shuffling scheme that DP-2Stage modifies."},{"cited_title":"Deep learning with differential privacy","cited_arxiv_id":null,"evidence_quote":"Supplies DP-SGD, the private optimizer used in stage two and in the DP-Standard and DP baseline comparisons."},{"cited_title":"The algorithmic foundations of differential privacy","cited_arxiv_id":null,"evidence_quote":"Supplies the definition of (ε,δ)-differential privacy and the Gaussian mechanism that ground the privacy model."},{"cited_title":"Synthetic data--anonymisation groundhog day","cited_arxiv_id":null,"evidence_quote":"Provides the preprocessed Texas Hospital Discharge dataset used as one of the three evaluation datasets."},{"cited_title":"Aim: an adaptive and iterative mechanism for differentially private synthetic data","cited_arxiv_id":null,"evidence_quote":"Supplies AIM, the marginal-based DP baseline that the appendix compares the LLM methods against."}],"review_version":1}