{"id":"79ff5ac6-5cfe-4ee3-a5a2-2c0b9219ff2b","arxiv_id":"2509.10696","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Struct-Bench is a Context-Free Grammar based evaluation platform showing that current differentially private synthetic text generators often fail to preserve structure and semantic diversity.","lead":"This paper introduces Struct-Bench, a benchmark for evaluating synthetic text generated from structured private data, such as customer records with natural language fields. It tests whether differentially private generation methods preserve both the format and the meaning of the original data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KND's cosine-similarity proxy may be insensitive to logical consistency, undercutting the structural-vs-semantic tradeoff claim on benchmarks like Grounding.","rationale":"I read the paper in good faith: Struct-Bench is a well-specified, publicly released benchmark with a plausible suite of metrics, and the reader's conditional verdict is appropriate. My concern is not that the framework is unusable, but that its headline empirical finding about structure-versus-semantics is only as strong as the KND metric. The reader identified the user-provided CFG and KND as the weakest assumptions; I partially agree. My specific worry is sharper: KND's embedding-cosine core may be insensitive to the logical consistency and entailment-type dependencies that are central to several benchmark datasets, especially Grounding (source1/source2 consistency) and Reviews (text/sentiment consistency). The paper explicitly acknowledges in Section 2 that CSGs would capture semantic dependencies but that it chooses KND for practicality; no validation of KND's sensitivity to the actual dependency is provided. The case study's reformatting trick further shows that CFG-PR can be improved without improving the underlying data semantics, which the authors themselves illustrate in Figure 5. If my proposed test shows KND is sensitive to consistency, the concern dissolves and the benchmark's claims stand. If the test fails, the paper's contribution should be reframed: Struct-Bench would still be a useful multi-metric evaluation platform, but the 'semantic performance' component would need to be interpreted as topical diversity/similarity, not logical fidelity, and the SOTA-limitation claim would need qualification. Given that the reader already recommends CONDITIONAL acceptance, I do not see a reason to change the verdict; the concern reinforces the condition rather than overturning the contribution.","tokens_in":24970,"tokens_out":6252,"duration_ms":59406,"concrete_test":"On the Grounding dataset, shuffle source2 across samples while keeping the marginal topic and token-length distributions unchanged, then recompute KND((source1, source2)) between the original real dataset and the shuffled synthetic dataset and compare with the real-to-real baseline value. If the W2 distance does not increase substantially, KND is insensitive to the consistency dependency. Independent validation: sample 200 real and synthetic (source1, source2) pairs, have GPT-4o or a human annotator label whether the sources are consistent, and compute the rank correlation between the consistency label and embedding cosine similarity; if the correlation is near zero, the KND metric does not measure what it claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Struct-Bench's central finding that SOTA DP methods cannot capture structure without sacrificing semantic performance rests on the assumption (Section 2.1, Appendix A.1) that KND, defined as Wasserstein-2 distance between distributions of cosine similarities of key-node pair embeddings, measures the semantically meaningful dependencies between nodes. Cosine similarity is a topical-similarity measure, not a logical-consistency measure. On the Grounding dataset (Table 5), the key node pair is (source1, source2), and the generated data deliberately varies whether the two sources are consistent or inconsistent. Contradictory sources about the same entity can have cosine similarity as high as consistent ones, so a generator that outputs topically similar but internally contradictory sources can match the real cosine-similarity distribution while destroying the consistency property that a practitioner cares about. The same concern applies to (text, sentiment) pairs in Reviews, where positive and negative reviews of the same product are topically close. The paper itself notes in Section 2 that a CSG would be needed to capture semantic dependencies as hard constraints, and chooses KND empirically; however, no sensitivity or correlation check against human/LLM judgments is reported. Additionally, CFG-PR is a formatting-compliance score that the case study inflates by post-hoc reformatting (Section 4, Figure 5), so 'structure' in the headline tradeoff may not be the structure that matters downstream. The benchmark is a useful contribution, but the specific claim that current methods sacrifice semantic performance is only as strong as the KND proxy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Struct-Bench, an evaluation framework and benchmark for differentially private (DP) synthetic data generation on structured datasets that contain natural language fields. The framework requires a user-supplied Context-Free Grammar (CFG) to parse each sample into nodes, and it defines three classes of metrics: structural (CFG Pass Rate, Key Node Dependency, Attribute Match), non-structural (KNN-Precision/Recall), and downstream task accuracy (TSTR). The benchmark includes seven datasets (ShareGPT, ICLR 2024 reviews, Water, Arena, Adult, Synthetic Reviews, Synthetic Grounding), each annotated with a CFG and key-node pairs. The authors benchmark several DP generators (Private Evolution, Instruction Following, DP fine-tuning, and non-private fine-tuning variants) under various privacy budgets, and report that no single metric fully describes synthetic data quality and that existing SOTA methods fail to capture structure without sacrificing semantic performance. A case study then uses Struct-Bench to improve Private Evolution via LLM-assisted CFG reformatting, node extraction and auto-generation, and format-token fixing, reporting improved CFG-PR, KNN-Precision, KNN-Recall, and several attribute-match metrics.","tokens_in":25393,"tokens_out":9719,"duration_ms":81134,"significance":"If the technical concerns are resolved, Struct-Bench would be a valuable community resource: it is, to my knowledge, the first benchmark focused specifically on DP synthetic data for structured datasets with natural-language components. The paper ships a public implementation and leaderboard, provides CFG annotations for seven diverse datasets, and makes a credible case that structure and content quality are orthogonal and need to be measured together. The multi-metric analysis is a useful correction to single-number evaluations. The case study also demonstrates a concrete workflow for using the benchmark to drive algorithm development. The main threats to this significance are the unvalidated and partly duplicate Key Node Dependency metric, the asymmetric foundation-model choices in the headline method comparison, and the incorrect or incomplete privacy accounting for DP-SGD with δ=0.","major_comments":[{"comment":"The paper states in Section 3.2 that all experiments set δ=0. Since DP-FT is implemented with DP-SGD, which relies on Gaussian noise, an (ε,0)-DP guarantee is not achievable with the standard mechanism; either the privacy accounting is incorrect or the statement omits the actual δ value (e.g., the moments accountant result). Please specify the exact mechanism, the privacy accountant, and the realized δ per dataset, or rerun the DP-FT experiments with correctly accounting for δ > 0. This is necessary for the benchmark's privacy claims to be valid.","section":"Section 3.2, privacy accounting"},{"comment":"KND is defined as the Wasserstein-2 distance between distributions of cosine similarities of embeddings of a key node pair (Oi, Oj). Cosine similarity is symmetric, so the distributions for (query, response) and (response, query) pairs are identical; the benchmark nonetheless reports them as separate metrics in Table 5 and in Figures 9, 15, 17, 20, and 21. Please either replace the scalar cosine similarity with an asymmetric dependency function (e.g., directional conditional likelihood) or remove the duplicated pairs, since the current metric suite overstates the number of independent signals.","section":"Section 2.1, Eq. (KND) / Table 5"},{"comment":"The headline finding that DP methods cannot preserve structure without sacrificing semantics rests on KND as a measure of semantic dependency. On the Synthetic Grounding dataset, the key node pair (source1, source2) is designed to vary in logical consistency, but cosine similarity captures topical similarity rather than logical consistency, so contradictory sources can have cosine similarity distributions close to consistent ones. The same issue arises for (text, sentiment) in Reviews, where positive and negative reviews are topically close. Section 2 explicitly notes that a context-sensitive grammar would be needed for semantic dependencies, yet no validation (e.g., correlation of KND with human or LLM judgments on any of the seven datasets) is provided. Please add a validation study or sharply qualify the semantic-structure conclusions.","section":"Section 2.1 / Appendix A.1"},{"comment":"In Table 6, PE and IF are run with GPT-4o while FT and DP-FT are limited to GPT-2, confounding method with foundation model. The conclusion that 'none of the existing SOTA DP methods are able to reliably capture the structural properties of data without sacrificing semantic performance' is therefore not directly supported as a statement about DP techniques. The authors already acknowledge this confound in the last bullet of Section 3.3, but the abstract and conclusion should be rephrased, and ideally same-model comparisons (e.g., PE with GPT-2 for all datasets, or DP-FT with a larger open-weights model) should be reported.","section":"Table 6 and Section 3.3"},{"comment":"No variance estimates are reported for the main benchmark results. Given the stochasticity of DP noise, LLM sampling, and SGD fine-tuning, point estimates without error bars cannot support small quantitative comparisons (e.g., CFG-PR 0.17 vs 0.19 on ICLR, KND differences of 0.02). Please report means and standard deviations over at least three independent runs for the central tables, or explicitly state that the numbers are from a single run and soften the corresponding comparative claims.","section":"Tables 2 and 6"},{"comment":"The benchmark metrics are not fully reproducible from the manuscript because key implementation details are missing: the embedding model and k value for KNN-Precision/KNN-Recall, the embedding model for KND, the number of samples used to estimate the Wasserstein-2 distances, and the LLM/prompts used for AM semantic attributes such as topic and intent are not specified. Please add a complete specification in the appendix, even if the code is public.","section":"Section 2.1 and Appendix A"}],"minor_comments":[{"comment":"The sentence 'KNN-Precision of vanilla PE increases from 0.56 to 0.69' appears twice in succession; please remove the duplicate.","section":"Section 4, Problem 2"},{"comment":"The label 'KNS' in Figure 16 should presumably be 'KND'.","section":"Section D.4, Figure 16"},{"comment":"The dataset names in Figure 2 are garbled (e.g., '6KDUH*37'), seemingly because of a font encoding problem; please regenerate the figure with standard text rendering.","section":"Figure 2"},{"comment":"The heading refers to Llama2-7b; please verify the model name/version and use consistent spelling.","section":"Section C.3"},{"comment":"Please specify whether the semantic attributes used in AM (topic, intent, sentiment) are obtained from GPT-4o or another labeler, and include the exact labeling prompt, since these are part of the benchmark protocol.","section":"Section B.4 / Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a timely and practically important gap, and the public codebase/leaderboard are valuable. The main risks are the privacy accounting for DP-FT (δ=0 with DP-SGD is not credible), the unvalidated and partly duplicate KND metric, and the confounded model-method comparison in the headline results. If these are fixed, the benchmark could be a useful community resource; I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. Struct-Bench fills a real gap: evaluating DP synthetic data that has both structure and natural language fields. The CFG-based representation is a simple, workable idea, and the benchmark ships seven datasets, reference metrics, and a leaderboard. That's a concrete contribution to a subfield that mostly evaluates on unstructured text or pure tabular data.\n\nWhat's new is the evaluation protocol, not the generation methods. CFG-PR is a straightforward but effective way to score structural validity, and KND is a plausible attempt to measure node-pair dependencies without hand-crafted constraints. The multi-metric approach is sensible—the results do show that a single number misses important failures, e.g., high semantic quality with zero structural compliance. The public release and leaderboard make it easy to adopt.\n\nThe soft spots are mostly about the strength of the empirical claims. First, the main comparisons pit GPT-2-based DP-FT against GPT-4o-based PE/IF. That confounds the method with the model. The Instruct DP-FT results partially address this, but they still don't use the same foundation model across all baselines. Second, the central tables (Tables 2 and 6) lack error bars. Some gaps look meaningful, but with sample sizes in the hundreds and presumably three runs, we don't know. Third, KND's reliance on embedding cosine similarity is a real limitation. On the Grounding dataset, the key relation is whether two sources are logically consistent; cosine similarity can't distinguish a contradiction from a paraphrase. The stress-test note has this right. The paper acknowledges the CSG alternative but doesn't validate KND against human judgments or a task-based measure. That weakens the headline claim that current methods 'sacrifice semantic performance'—on some datasets, KND may simply be measuring topical similarity, not the dependency that matters.\n\nThe case study is useful but self-referential in an open way: reformatting raises CFG-PR by construction, and node extraction targets KNN-Recall. I don't see that as a flaw in the benchmark; it's the intended workflow. Just don't read the case study as an independent evaluation.\n\nCitation pattern is fine; the related work is on point, and self-citations are to the PE line they build on.\n\nBottom line: this is a solid benchmark paper, worth serious refereeing. The framework is reusable and the datasets are non-trivial. I'd want the authors to fix the model pairing, add variance estimates, and either validate KND or soften the 'semantic performance' claim. The benchmark itself deserves to exist.","headline":"A genuinely useful benchmark for DP structured text generation with a clean CFG-based evaluation protocol; the empirical claims need error bars and a fairer model pairing, and the KND proxy deserves validation.","tokens_in":25823,"tokens_out":2114,"would_cite":true,"duration_ms":18392,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A grammar-based benchmark shows DP text generators can't keep structure and semantics together.","keywords":["differential privacy","synthetic data generation","structured text","context-free grammar","evaluation benchmark","Private Evolution","semantic diversity","TSTR evaluation"],"falsifier":"Run a privacy-preserving generator at $\\epsilon=4$ on the ICLR dataset and check whether it can simultaneously reach a CFG pass rate above $0.95$ and a KNN-recall above $0.8$ under the paper's grammar and embedding setup; a single such result would refute the claim that current DP generators cannot preserve structure and semantics together.","tokens_in":1652,"feed_emoji":"📊","tokens_out":2281,"duration_ms":87984,"temperature":0.7,"pith_summary":"Struct-Bench is a benchmark and evaluation protocol for differentially private (DP) synthetic data generated from structured datasets whose fields contain natural language. The authors aim to show that a single quality metric cannot describe such synthetic data, and that current state-of-the-art DP generators cannot capture the dataset's structural constraints without sacrificing semantic performance. The protocol works by having the user encode the dataset's expected shape as a context-free grammar, then scoring the synthetic data on structural compliance, node-level and sample-level attribute matches, semantic precision and recall, and downstream label-prediction accuracy. On seven datasets, including conversation logs, paper reviews, product reviews, and tabular records, the benchmark exposes tradeoffs that simpler metrics hide. The paper also shows how its metric profiles guided modifications to Private Evolution that improve structural validity and semantic diversity at no additional privacy cost.","feed_headline":"Benchmark exposes a structure-versus-meaning tradeoff in DP text","feed_subtitle":"A grammar-based scorecard shows where DP generators lose structure and where they lose meaning.","key_machinery":"The load-bearing mechanism is the user-supplied context-free grammar (CFG) that encodes the dataset's structural schema. Every sample, real or synthetic, is parsed with this grammar into typed nodes; a tree-pattern tool locates designated key-node pairs, and the framework then compares real and synthetic distributions of node dependencies and attributes using Wasserstein-2 or total-variation distances. The CFG makes structure a checkable formal property: it lets the benchmark measure 'did the output follow the shape of the data' separately from 'is the content semantically good'.","core_discovery":"The central discovery reported here is two-sided. First, no single evaluation metric fully describes the quality of DP synthetic data for structured, natural-language datasets: a sample that is semantically fluent can be structurally invalid, and a sample that satisfies a grammar can be semantically hollow. The paper demonstrates this by separating structural metrics from content metrics, and shows, for example, that instruction-following and Private Evolution reach high CFG pass rates on tabular and synthetic datasets while their semantic diversity (KNN-recall) stays near zero. Second, none of the state-of-the-art DP generators evaluated—Private Evolution, instruction following, and DP fine-tuning—reliably captures structure without sacrificing semantic performance; DP fine-tuning in particular achieves a CFG pass rate of zero on all datasets at $\\epsilon=4$. The authors state this as a motivation for multi-metric benchmarking and for further research on structure-aware privacy-preserving generation.","pith_inferences":["The paper stops short of claiming that CFG-based structure is sufficient for all structured text; a natural extension is to test whether context-sensitive constraints that a CFG cannot express, such as long-range agreement between non-adjacent nodes, produce failures that the benchmark's current metrics under-report.","Because the benchmark treats the user-provided grammar as a given, a useful next experiment would be to vary the grammar itself: if two equally reasonable grammars for the same dataset rank the same generators differently, then benchmark conclusions are grammar-sensitive and should be reported together with the grammar.","The metric suite could be reused outside differential privacy, for example to evaluate synthetic training data for data augmentation or conditional generation, where structural fidelity and semantic diversity are also the properties that matter; the paper mentions this possibility only in passing.","One could add a privacy side to the benchmark, for example measuring how much verbatim memorization of private samples survives in high-scoring synthetic outputs, to check that high quality scores do not come from copying; the current metrics do not assess this."],"forward_implications":["If synthetic text is meant for downstream pipelines, structural validity has to be a first-class evaluation axis, because a high single-similarity score can mask output that no parser or downstream utility would accept.","DP methods that rely on fine-tuning small open models are unlikely, on their own, to produce schema-compliant structured text; instruction-guided generation or training-free methods such as Private Evolution are needed to reach usable pass rates.","The benchmark's metric profile can be read diagnostically: low CFG pass rate points to formatting or schema loss, low KNN-recall points to diversity loss, and high key-node-dependency distance points to broken semantic correlations.","The described modifications to Private Evolution—reformatting invalid samples after private voting and extracting one node type while auto-generating the rest—are direct recipes for improving both structural validity and semantic diversity without extra privacy cost under the post-processing property.","A common multi-metric protocol would make DP structured-text generators comparable on the same yardstick, potentially shifting research incentives from single-score optimization toward balanced improvement."],"supporting_citations":[{"why":"It introduces Private Evolution, the training-free DP generation method that the paper benchmarks and later modifies.","marker":"[27]"},{"why":"It supplies the Augmented Private Evolution text-generation variant and the DP text setting used in all PE experiments.","marker":"[60]"},{"why":"It provides DP fine-tuning of language models, one of the central baselines compared here.","marker":"[63]"},{"why":"It supplies the improved precision and recall definitions that underlie KNN-Precision and KNN-Recall.","marker":"[25]"},{"why":"It provides the tree-pattern matching tool used to locate key-node pairs for the Key Node Dependency metric.","marker":"[26]"},{"why":"It provides Longformer, the downstream model used for label-prediction evaluations.","marker":"[7]"},{"why":"It establishes prior benchmarking protocol for DP synthetic data that the paper follows for non-structural metrics.","marker":"[51]"}],"fun_headline_variants":["DP text benchmark: no metric captures structure and meaning","Grammar vs. semantics: DP generators fail on both","Struct-Bench: where DP text drops structure or meaning","Benchmark shows DP text trades grammar for meaning"],"cache_read_input_tokens":27904,"weakest_assumption_plain":"The benchmark's conclusions rest on the assumption that a user-provided context-free grammar captures the structural properties that matter for downstream use, and that embedding cosine similarity faithfully measures semantic dependency; if either fails, the rankings the benchmark produces could mislead.","fun_headline_variants_meta":{"raw":{"variants":["DP text benchmark: no metric captures structure and meaning","Grammar vs. semantics: DP generators fail on both","Struct-Bench: where DP text drops structure or meaning","Benchmark shows DP text trades grammar for meaning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1415,"prompt_tokens":969,"completion_tokens":446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":383}},"tokens_in":585,"tokens_out":446,"duration_ms":4136,"temperature":1.0,"reasoning_tokens":383,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:52:47.660550+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a privacy-preserving generator at $\\epsilon=4$ on the ICLR dataset and check whether it can simultaneously reach a CFG pass rate above $0.95$ and a KNN-recall above $0.8$ under the paper's grammar and embedding setup; a single such result would refute the claim that current DP generators cannot preserve structure and semantics together.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It introduces Private Evolution, the training-free DP generation method that the paper benchmarks and later modifies."},{"cited_title":"Kynkäänniemi, T","cited_arxiv_id":null,"evidence_quote":"It supplies the improved precision and recall definitions that underlie KNN-Precision and KNN-Recall."},{"cited_title":"Tregex and tsurgeon: Tools for querying and manipulating tree data structures","cited_arxiv_id":null,"evidence_quote":"It provides the tree-pattern matching tool used to locate key-node pairs for the Key Node Dependency metric."}],"review_version":1}