{"id":"3ea9a295-5a98-450b-80b9-6d0f916b38d4","arxiv_id":"2505.18485","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Injecting domain knowledge into prompts can substitute for many in-context examples in LLM-based synthetic tabular data generation, cutting required example counts by 40-90%.","lead":"Large language models can generate fake tabular data, but usually need many real example rows to copy the shape of the data. This paper shows that adding a short English description of the data's known properties, such as ranges or that the points look like a dinosaur, can cut the number of example rows by 40-90% while keeping quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 40–90% ICL reductions are derived from only two ICL sizes per dataset with no seed variance or error bars, so the central quantitative claim is underdetermined.","rationale":"The paper's central contribution is a quantitative trade-off between knowledge prompts and ICL examples, supporting both the abstract's scaling-law claim and the practical promise of token savings. My stress-test focuses on how the numbers are measured. Section 4.2 describes only two ICL sizes for each dataset, and the text reports figures such as Dino 40%, Away 90%, and High Lines 80% with no standard errors. Since generation is stochastic and ICL subsets are random, a two-point comparison cannot separate signal from noise, so the reduction percentages are not identified. This is more load-bearing than the reader's hand-selection concern, because even if knowledge phrases were obtained automatically, the claimed magnitudes would still lack statistical support. The reader's external-validity concern remains valid and complementary. The taxonomy of knowledge types and the OOD generalization experiments are constructive and could form the basis of a solid conditional paper. However, the strongest claim as written requires either more measurement points with variance or a clear restriction of scope to a demonstration on selected datasets with human-authored knowledge. I therefore keep the verdict at CONDITIONAL (no change) rather than rejecting, because the concern is addressable with a straightforward experimental rerun.","tokens_in":924,"tokens_out":1785,"duration_ms":58892,"concrete_test":"Re-run RQ1 on the Dino, Away, and High Lines datasets using ICL sizes {5,10,20,30,50,100}, at least 10 random ICL subsets per size, and at least 5 generation seeds per condition. Record Hausdorff distance for Datasaurus and MAPE for AP Calculus under No-KGP and Semantic-KGP, then fit a monotone curve with uncertainty (e.g., isotonic regression or a Gaussian process with bootstrap). Compute the distribution of the horizontal difference needed to match quality between the two conditions. If the claimed 40-90% reductions fall outside the 95% bootstrap interval, or if the endpoint-only estimates shift by more than 20 percentage points, the central quantitative claim is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 reports the central quantitative finding: KGP reduces ICL examples by 40-90% in the easy data context and 50-80% in the hard data context. The underlying experiments use only two ICL sizes per dataset, 20 vs 50 for AP Calculus and 10 vs 100 for Datasaurus, with no repeated random ICL draws, no generation seeds, and no stated interpolation or extrapolation model. A 90% reduction on the Away dataset is inferred by matching one Semantic-KGP endpoint to one No-KGP endpoint; with only two points per curve, any monotone or non-monotone behavior between endpoints is consistent with the plot, and sampling noise is uncontrolled. The abstract also promises an empirical scaling law, but no equation, fitted parameters, or goodness-of-fit appears in Sections 4 or 5. The hand-selection of knowledge phrases is a separate external-validity threat; the more basic issue is that, even internally, the claimed magnitudes are not statistically identified by the reported design.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes knowledge-guided prompting (KGP), a prompt optimization strategy that augments LLM prompts for synthetic tabular data generation with statistical, semantic, and symbolic domain knowledge. The central claim is that KGP can substantially reduce the number of in-context learning (ICL) examples needed to reach a given generation quality, with reported reductions of 40–90% in 'easy' and 50–80% in 'hard' data contexts, and an 'empirical scaling law' relating quality to knowledge level and example count. The experiments cover AP Calculus textbook functions, Datasaurus Dozen datasets, and an O2-sensing cyber-physical dataset, using metrics such as MSE, Hausdorff distance, NLL, KL divergence, DCR, and MLU. The paper also studies out-of-distribution generation and a noise-and-refix case study.","tokens_in":13169,"tokens_out":10117,"duration_ms":70524,"significance":"The question at the heart of the paper is important: if a few English sentences about the target distribution can substitute for dozens of example rows, KGP would reduce token costs and help low-data generation, and the proposed taxonomy of strong versus weak knowledge is a useful organizing device. The paper does contain some concrete positive results, such as Semantic KGP reducing the sigmoid OOD MSE from 0.11 to 0.002, and the visual comparison in Figure 2a shows a large Hausdorff improvement at the sampled point. However, the central quantitative claims, the scaling law, and the summary statements are not supported by the evidence as reported; several self-reported tables (Tables 3, 4, 6, 7) contain negative or contradictory cases. Because the core idea is plausible but the load-bearing evidence is underdetermined, the manuscript needs major revision rather than acceptance.","major_comments":[{"comment":"The promised 'empirical scaling law' is never formulated. The abstract and conclusion claim a scaling law that quantifies generation quality as a function of domain knowledge and example count, but Sections 4 and 5 contain no equation, no fitted parameters, no goodness-of-fit, and no stated functional form. The 40–90% reduction statements in §4.2 are descriptive comparisons of the plotted curves, not predictions from a scaling law. The authors should either state the law explicitly with fitted parameters and uncertainty, or remove the scaling-law claim from the abstract and conclusion.","section":"Abstract, §4.2, §5"},{"comment":"The headline reductions are underdetermined by the reported design. For AP Calculus only two ICL sizes (20 and 50) are evaluated, and for Datasaurus only two (10 and 100), with no repeated random ICL draws, no generation seeds, and no error bars. The 90% reduction on the Away dataset is inferred by matching one Semantic-KGP endpoint to one No-KGP endpoint; with two points per curve, any monotone or non-monotone behavior between the endpoints is consistent with the plot. In addition, the Figure 2 caption refers to '60 In-Context Samples' while §4.2 describes 20 and 50; this inconsistency must be resolved. The authors need more ICL sizes and repeated trials before the quantitative reduction claims can be supported.","section":"§4.2, Figure 2"},{"comment":"The concentric nesting of knowledge types (Semantic = Statistical + Semantic; Symbolic = all three) means the comparison in Table 4 is not an ablation. The improvement attributed to 'Semantic KGP' over 'Statistical KGP' could be due entirely to the additional statistical range information, and 'Symbolic KGP' is always evaluated with statistical and semantic knowledge present. To answer RQ3, the authors need a condition in which only the semantic description is added, or they must re-frame the claims as comparisons of composite prompts.","section":"§4.1 'KGP Scope'"},{"comment":"The semantic KGP phrases are hand-picked per dataset with knowledge of the target shape (e.g., 'x and y coordinates of points when plotted visually depict a dinosaur', 'high lines'), and the statistical ranges in Table 2 are the exact ranges of the target datasets. No automatic knowledge-extraction method or held-out evaluation of the knowledge phrases is provided. The claimed 40–90% ICL reductions therefore conflate the effect of generalizable domain knowledge with the effect of test-set-specific hints, so the reductions may not transfer to new datasets. The authors should either supply an automatic extraction protocol, evaluate on datasets whose knowledge phrases were produced without inspecting the target distribution, or explicitly restrict the claim to the 'knowledge available' setting and discuss the leakage caveat.","section":"Table 2, §4.2"},{"comment":"The OOD generalization claim is selective. For Bohachevsky, Statistical KGP increases MSE from 1.62 to 2.23, while only Semantic KGP improves it to 0.44; the text reports only the Semantic improvement. The '78% to 90% lower error' summary also does not match the table's 98% (Sigmoid) and 73% (Bohachevsky). The authors should report both positive and negative directions and attribute the improvement to the specific knowledge type rather than to KGP as a whole.","section":"§4.3, Table 3"},{"comment":"Table 4 contradicts the claim in §4.2 that 'incorporating any form of KGP ... leads to a significant reduction in data generation error.' For y = x + 2 sin(x), Symbolic KGP produces MSE 0.57 versus 0.40 without KGP; for y = 2x^3 - 15x^2 + 36x the differences are within rounding. The manuscript's own Table 4 therefore shows that knowledge guidance is not uniformly beneficial, and the text should acknowledge and analyze these negative cases rather than only reporting the reductions.","section":"§4.4, Table 4"},{"comment":"The overall finding that KGP achieves 'optimal performance across all three standard synthetic table metrics, with an average enhancement of 50%' is not supported by the tables. In Table 7, Semantic KGP worsens linear-regression MLU for AP Calculus (2.13 vs 1.61); in Table 6, AP Calculus DCR is 0 for all conditions, which the table caption itself identifies as an unacceptable data leak. The summary should be based on all rows of the tables, including the cases where KGP hurts or provides no benefit.","section":"§4.5, Tables 6–7"}],"minor_comments":[{"comment":"There are numerous typos and misspellings, including 'Hustoff' for 'Hausdorff' in Figure 2, 'A way' for 'Away', 'Stastical' in §3.1, 'inconsistensies' in §3, 'taasks' in §1, and 'Dinausour' in Figure 5.","section":"Throughout"},{"comment":"The Figure 2 caption mentions 'MAPE' and 'Hustoff distance', but MAPE is not defined in the metrics list in §4.1, and the main text describes only Hausdorff distance and MSE. The figure and text should be reconciled.","section":"§4.1, Figure 2"},{"comment":"The phrase 'Figure 2a,b,c and e' for simple distributions and 'Figure 2c and 2f' for complex distributions is confusing: Figure 2 appears to contain six subfigures, and the text does not clearly map each dataset to a subfigure. Please make the subfigure references explicit.","section":"§4.2"},{"comment":"Key experimental details are missing: the chunk size k and number of chunks c in the pipeline are never specified, and the generation temperature, model version/date, number of sampled rows per chunk, and number of runs are not reported. These details are needed for reproducibility.","section":"§3, §4.1"},{"comment":"The arrows in Table 4 are ambiguous: it is unclear whether each arrow compares the current column to the previous column or to the 'W/o KGP' column, and the '∼' symbol is not defined in the caption.","section":"Table 4"},{"comment":"Since the paper is about the tradeoff between prompts and ICL examples, it should quantitatively compare or at least discuss the findings of [19] in the tabular data setting, rather than only citing it.","section":"References"},{"comment":"The 'noise-and-refix' procedure is introduced without specification; please describe how noise is added, how the LLM is prompted to fix it, and how the resulting synthetic data are evaluated against the original ground truth.","section":"§4.6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from a clearer separation of the 'knowledge available' setting (where knowledge phrases are allowed to be hand-crafted) from the 'knowledge inferred' setting (where they are produced automatically); currently the abstract and introduction imply the latter while the experiments use the former. The selective reporting in Tables 3 and 7 is a serious presentation problem and should be addressed head-on in revision. If the authors can provide the full experimental protocol, including seeds, ICL draws, and a formulated scaling law, the central claims could become testable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe paper has a genuinely useful idea: inject domain knowledge into the prompt to reduce how many in-context examples you need for LLM-based tabular data synthesis. That is a legitimate extension of the Le Scao and Rush prompt-worthiness result from NLP classification to joint-distribution generation, and the symbolic/semantic/statistical taxonomy is a handy organizing device. I also give them credit for including OOD generalization and for showing that misleading knowledge can hurt—that is a real honesty signal.\n\nBut the headline numbers are not supported by the design. The 40–90% and 50–80% reduction figures come from comparing two ICL sizes per dataset (e.g., 20 vs 50, 10 vs 100) with no repeated draws, no seeds, no error bars on the key curves. With two points, any monotone curve between them is consistent; you cannot identify scaling behavior. The 'empirical scaling law' promised in the abstract never appears as an equation or fitted model. The semantic knowledge phrases are hand-picked after looking at the target shape, so part of the gain is likely self-fulfilling. Some internal numbers smell off too: Table 4 shows Symbolic KGP making things worse on two of four functions (e.g., 0.40 → 0.57 for x+2sin(x)), which sits awkwardly next to 'symbolic knowledge is strong and beneficial.' The OOD improvements are single measurements, and the noise-and-refix case study has no error analysis.\n\nNone of this kills the core idea. I suspect KGP does help in low-data regimes, and the paper is a reasonable first empirical cut. But the central quantitative claim is underdetermined, and the scaling-law framing overstates what was measured.\n\nWorth sending to peer review? Yes. A good referee can push for more ICL sizes, repeated seeds, sensitivity of the semantic phrases, and an actual scaling-law formula. I would not cite it in its current form, but I'd want to see the revised version. It would make a decent reading-group piece on how to (and how not to) quantify prompt-vs-example tradeoffs.","headline":"Useful idea, under-supported headline numbers: the KGP reductions rest on two-point curves with no error bars, and the promised scaling law is never formulated.","tokens_in":13507,"tokens_out":2115,"would_cite":false,"duration_ms":17802,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Knowledge written into a prompt substitutes for many in-context examples in LLM synthetic tabular data generation.","keywords":["knowledge-guided prompting","in-context learning","synthetic tabular data","large language models","prompt optimization","scaling law","out-of-distribution generation","data augmentation"],"falsifier":"Run KGP on a held-out dataset using a knowledge statement produced automatically from the data (e.g., extracted column ranges and relationships) rather than one written after inspecting the data. If the 40–90% ICL reduction does not reproduce, the effect is driven by test-set-informed prompt selection, not by knowledge injection per se.","tokens_in":12614,"feed_emoji":"📊","tokens_out":3615,"duration_ms":25265,"temperature":0.7,"pith_summary":"The paper claims that explicit domain knowledge injected into a prompt can substitute for many in-context example rows when an LLM generates synthetic tabular data. On simple joint distributions, knowledge-guided prompting matches the quality of example-only prompting with 40–90% fewer examples; on hard distributions, the reduction is 50–80%. The authors propose an empirical scaling law that relates generation quality to the amount of injected knowledge and the number of examples. A reader should care because token budgets and scarce real data are the main bottlenecks in LLM-based synthetic data generation.","feed_headline":"Knowledge prompts replace 40–90% of data examples","feed_subtitle":"Injecting domain knowledge into LLM prompts for synthetic tables matches quality with far fewer in-context samples.","key_machinery":"The central mechanism is the knowledge-guided prompt, an English-language statement of global data properties attached to each chunk of in-context examples. Three nested knowledge levels are used: statistical (column ranges), semantic (shape or dependency descriptions such as 'x and y coordinates depict a dinosaur'), and symbolic (the governing equation). The concentric construction means each higher level adds to the lower ones. The prompt carries the global knowledge, the chunks carry local knowledge, and together they condition the LLM's generation.","core_discovery":"The central claim is that in-context examples and domain knowledge are partially substitutable inputs for LLM-based tabular data generation. The paper defines knowledge-guided prompting (KGP) with three nested levels—statistical, semantic, and symbolic—and shows that adding knowledge improves generation quality at a fixed example count, and that the same quality can be reached with fewer examples. It reports that KGP also extends generation to out-of-distribution regions, with error reductions of 78–90% compared to no-knowledge prompting when exploring unobserved fields. The empirical scaling law states that quality varies with increasing domain knowledge and decreasing example count, providing a framework for predicting data requirements.","pith_inferences":["The reported savings likely depend on the quality and specificity of the English knowledge statements; automatic knowledge extraction from the target dataset would be the natural test of whether the scaling law holds outside hand-crafted prompts.","The framing suggests a broader principle for LLM prompting: declarative knowledge and exemplars are interchangeable currencies, which could extend beyond tabular data to code generation, planning, and other structured generation tasks.","A testable extension is to measure the token-equivalence rate of a knowledge statement—how many example rows one sentence replaces—as a function of model scale, which the paper does not address."],"forward_implications":["If the claim is correct, LLM-based synthetic data generation can operate with far fewer real examples, directly reducing token cost and easing data scarcity.","KGP can be combined with ICL rather than replacing it, giving practitioners a two-knob design space in which knowledge statements are traded against example counts.","The scaling law, if it generalizes, would let practitioners predict how many examples are needed once a knowledge level is chosen, and vice versa.","In noisy or sparse settings, knowledge guidance helps the LLM correct errors and explore unobserved regions, enabling safer augmentation of incomplete datasets."],"supporting_citations":[{"why":"Establishes the prompt-versus-example tradeoff that KGP extends to multiple knowledge levels and tabular data.","marker":"[19]"},{"why":"Provides the LLM-based tabular data generation baseline (GReaT) that motivates the ICL setting.","marker":"[4]"},{"why":"Shows how to transform table rows into text for LLM tasks, the encoding used throughout the paper.","marker":"[8]"},{"why":"Introduces prompting with domain requirements in low-data regimes, the closest prior approach to KGP.","marker":"[29]"},{"why":"Supplies the Datasaurus Dozen datasets whose identical statistics and varied shapes are used to test semantic knowledge.","marker":"[23]"},{"why":"Supplies the AP Calculus textbook datasets with symbolic equations and function-description knowledge.","marker":"[3]"}],"fun_headline_variants":["Prompt knowledge outweighs example count for tabular data","Fewer examples needed with knowledge-guided prompts","Domain knowledge in prompts cuts example needs by up to 90%","Knowledge prompts unlock synthetic data beyond example range"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed savings rest on the authors' manually chosen English knowledge statements, which are selected after inspecting the target data; if the knowledge selection itself is a form of fitting, the reductions may not transfer to new datasets.","fun_headline_variants_meta":{"raw":{"variants":["Prompt knowledge outweighs example count for tabular data","Fewer examples needed with knowledge-guided prompts","Domain knowledge in prompts cuts example needs by up to 90%","Knowledge prompts unlock synthetic data beyond example range"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001101,"raw_usage":{"total_tokens":4603,"prompt_tokens":962,"completion_tokens":3641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":3579}},"tokens_in":578,"tokens_out":3641,"duration_ms":19880,"temperature":1.0,"reasoning_tokens":3579,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:29:38.212252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run KGP on a held-out dataset using a knowledge statement produced automatically from the data (e.g., extracted column ranges and relationships) rather than one written after inspecting the data. If the 40–90% ICL reduction does not reproduce, the effect is driven by test-set-informed prompt selection, not by knowledge injection per se.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the prompt-versus-example tradeoff that KGP extends to multiple knowledge levels and tabular data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows how to transform table rows into text for LLM tasks, the encoding used throughout the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Datasaurus Dozen datasets whose identical statistics and varied shapes are used to test semantic knowledge."},{"cited_title":"Anton, I.C","cited_arxiv_id":null,"evidence_quote":"Supplies the AP Calculus textbook datasets with symbolic equations and function-description knowledge."}],"review_version":1}