{"id":"5d197c60-874f-42e9-ad56-b5c10adbc9ad","arxiv_id":"2504.20687","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Explainable AI applied to a real-versus-synthetic detection classifier pinpoints which features, dependencies, and value ranges a tabular generative model reproduces poorly.","lead":"This paper trains a classifier to tell real from synthetic tabular data, then uses explainable AI to show which features and combinations make synthetic rows detectable. The result is a diagnostic view of why a generator fails, beyond simple quality scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that XAI diagnostics reveal generator flaws is not validated against ground truth; explanations may describe detector artifacts rather than data-quality problems, and Section 6 already concedes this dependency.","rationale":"The paper is a clear demonstration of a plausible toolkit, with reproducible code, two datasets, and honest limitations. The central claim is not that the detector is a perfect quality oracle; the abstract says XAI reveals why synthetic data are distinguishable, which is literally what the detector explains. The load-bearing step is the next inference: that these explanations diagnose generator weaknesses and give deeper insight than standard metrics. That inference is untested. The paper's own Section 6 concedes the dependency on detection performance and sensitivity to irrelevant deviations such as numeric precision, so the concern is not speculative; it is an acknowledged gap in the argument. A controlled injected-flaw experiment would settle whether the explanations track real generator deficiencies or only classifier artifacts. Because the concern is real but addressable, and because the paper already positions itself as a demonstration rather than a definitive evaluation, the reader's CONDITIONAL verdict is appropriate; no adjustment is needed.","tokens_in":17258,"tokens_out":4008,"duration_ms":45066,"concrete_test":"Construct two controlled synthetic versions of the adult data by injecting known flaws: (i) shuffle the age-education_num dependency while preserving all marginals, and (ii) keep all dependencies intact but add the non-integer education_num artifact. Run the paper's full pipeline (tuned XGBoost plus PFI, TreeSHAP, PDP/ICE) on each version, together with standard metrics (alpha-precision, beta-recall, MMD, correlation-distance). A valid diagnostic should rank the injected dependency first in (i) and should not report dependency artifacts in (ii); if instead PFI or TreeSHAP flag the precision artifact as a top issue, the detector-proxy attribution fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 frames importance values, PDP/ICE curves, Shapley decompositions, and counterfactuals as diagnostics of what the generator was not able to reproduce realistically (Q1-Q4). The strongest claim, however, requires that these detector explanations can be attributed to generator flaws. Nothing in the experiments validates that attribution. The adult example interprets the periodic PDP drop for education_num as TabSyn generating non-integer values, but Section 6 admits the detector 'can be too sensitive and might detect deviations such as different numeric precision which might not be relevant to assess the practical utility of the synthetic data.' The same concern applies to PFI, which is computed by permuting features into off-manifold combinations (Section 4.2.1), and to TreeSHAP, whose path-dependent attribution reflects XGBoost's split structure. The evaluation never compares against a ground-truth generator flaw, never shows that repairing the highlighted issue improves a conventional fidelity or utility metric, and never runs the promised comparison to standard evaluation techniques. Absent such validation, 'what's wrong with the synthetic data' is operationally 'what the chosen detector exploits,' and the claimed depth beyond conventional metrics is an interpretation, not a demonstrated result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an evaluation framework for synthetic tabular data in which a binary classifier is trained to distinguish real from synthetic observations, and explainable AI methods are then applied to that classifier to explain why the synthetic data are distinguishable. The authors map four XAI tools to four diagnostic questions: permutation feature importance and Shapley values for challenging features and dependencies (Q1), partial dependence and ICE curves for unrealistic regions and over/underrepresented areas (Q2), Shapley decompositions for individual real and synthetic instances (Q3), and counterfactual explanations for minimal changes that would make a synthetic instance look real (Q4). The approach is demonstrated on the Adult dataset with TabSyn and, in an appendix, on the Nursery dataset with CTGAN. The central claim is that this XAI-on-detector pipeline provides deeper insight than conventional synthetic-data quality metrics and uncovers issues that standard evaluation techniques overlook.","tokens_in":17608,"tokens_out":3923,"duration_ms":44310,"significance":"If the central claim were fully established, the framework would be a practically useful addition to the synthetic-data auditing toolbox, because it moves beyond scalar quality scores toward feature- and instance-level diagnoses. The paper has real strengths: it gives a clean mapping from diagnostic questions to specific XAI tools, it uses a sensible detection model (tuned XGBoost) and compares several classifier families, it discusses the important distinction between marginal and conditional explanations, and it ships a public repository with scripts for tuning, training, and reproduction. The case studies are internally coherent and the XAI computations follow standard practice. However, the headline contribution is an empirical claim about explanatory depth relative to conventional metrics, and that claim is not yet backed by the experiments as written.","major_comments":[{"comment":"The abstract and Section 5 state that the approach 'uncovers issues overlooked by standard evaluation techniques,' but the experiments contain no comparison to conventional quality metrics such as maximum mean discrepancy, Wasserstein distance, precision/recall, or utility-based measures. Section 5.1 compares detection classifiers against each other, not against standard evaluation metrics, and Sections 5.2 and Appendix A present only the XAI-based diagnostics. Without a side-by-side demonstration that standard metrics fail to flag the education_num non-integer pattern, the occupation rare-class issue, or the class-dependency issue in Nursery, the claimed advantage over standard evaluation techniques is not demonstrated. I request an experiment that computes at least one or two standard fidelity/utility metrics on the same synthetic datasets and explicitly compares what those metrics indicate with what the XAI diagnostics indicate.","section":"Abstract; Section 5"},{"comment":"The pipeline attributes detector explanations to flaws of the generative model, but this attribution is assumed rather than validated. Section 6 itself concedes that the detector 'can be too sensitive and might detect deviations such as different numeric precision which might not be relevant to assess the practical utility of the synthetic data.' The Adult example interprets the periodic PDP dips for education_num as evidence that TabSyn generates non-integer values; this is plausible, but nothing in the experiments verifies that the highlighted pattern is a generator flaw rather than a detector artifact. The paper would be substantially strengthened by a validation study, for example by injecting controlled flaws into a generator and checking whether the XAI diagnostics recover them, or by repairing a diagnosed issue (e.g., rounding education_num to integers) and showing that a conventional fidelity or utility metric improves. Without such validation, 'what is wrong with the synthetic data' is operationally only 'what the chosen detector exploits.'","section":"Section 6; Section 4.2.2"},{"comment":"The feature-level conclusions are based on a single detector family (XGBoost), but PFI and TreeSHAP are known to reflect the model's internal structure as well as the data distribution. Section 5.1 shows that XGBoost has higher detection accuracy than logistic regression and random forest, but it does not examine whether the explanations are stable across detector choices. Since the central claim is that the XAI diagnostics reveal generator weaknesses rather than detector artifacts, I ask for a sensitivity analysis in which at least the global importance ranking and the PDP for education_num are recomputed with a different high-performing detector (e.g., a tuned random forest or a regularized logistic regression with engineered features) and the agreement or disagreement of the diagnostic conclusions is reported.","section":"Section 4.2.1; Section 5.2.1"}],"minor_comments":[{"comment":"The class label 'unknwon' on the y-axis appears to be a typo for 'unknown.'","section":"Figure 3b"},{"comment":"The text reads '5× 105 Monte Carlo samples'; this should presumably be 5×10^5 (or 500,000) Monte Carlo samples.","section":"Section 5.2.4"},{"comment":"The x-axis label 'degree 1 2' is ambiguous; it would be clearer to write 'Degree 1' and 'Degree 2' as separate legend entries or to explain the degrees in the caption.","section":"Figures 2b and 8b"},{"comment":"The conditional Shapley values are estimated with conditional inference trees, but the paper does not state the hyperparameter settings or the number of trees used for this estimator; a brief reproducibility note would be useful.","section":"Section 5.2.3"},{"comment":"The interpretation of the ICE curves as indicating 'low fidelity but high diversity' for education_num is somewhat compressed; since this is a key illustrative conclusion, one or two sentences explaining how the relative shares of real and synthetic ICE curves below/above 0.5 map to fidelity and diversity would improve readability.","section":"Section 5.2.2"}],"recommendation":"major_revision","confidential_remarks":"The framework is coherent and the code release is a real asset. The main revision burden is empirical: the paper must either add the missing comparison to standard metrics and the attribution validation, or substantially soften the claims in the abstract and Section 5. I do not see a fundamental flaw in the methodology itself, so rejection would be too strong; the requested experiments are within the scope of a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful paper, not a breakthrough. It takes the classifier-two-sample idea from Lopez-Paz and Zein and builds a proper diagnostic workflow around it — PFI, PDP/ICE, TreeSHAP interaction values, conditional Shapley, and MCCE counterfactuals — applied to XGBoost detectors on adult and nursery data with TabSyn and CTGAN. That integration is new and genuinely helpful for practitioners who currently stare at scalar metrics and want to know which features and value ranges are off. The writing is clear, the background is competent, and the authors are honest about the weak spots, including in Section 6 where they say the detector can be too sensitive and flag issues like numeric precision that don't matter for practical utility.\n\nWhat's strongest: the case studies illustrate the workflow concretely. The periodic PDP dips for education_num point to TabSyn generating non-integer values; the occupation plots show rare classes under- and overrepresented; the local Shapley analysis on the 17-year-old with education_num=4 gives a plausible diagnosis of an impossible dependency. The repo includes scripts for reproducing tables and figures, which is a real plus.\n\nThe soft spot is the one the authors also concede: the explanations describe the detector, not directly the generator. The paper never validates that the XAI findings correspond to actual data-quality problems. There is no experiment with a generator flaw introduced as ground truth, no check that fixing the highlighted issue improves a downstream utility metric, and no quantitative comparison to MMD, Wasserstein distance, or similar standard metrics. The stress-test is right that \"what's wrong with the synthetic data\" is operationally \"what the chosen detector exploits.\" That doesn't kill the paper—the authors' framing is mostly careful—but the abstract's claim that the approach \"uncovers issues overlooked by standard evaluation techniques\" is not backed by evidence in the paper. PFI and marginal/conditional Shapley have known off-manifold and dependency caveats; the paper acknowledges them, but the interpretation section doesn't weight them enough when drawing conclusions.\n\nWho is this for: practitioners building or auditing synthetic tabular data who want a debugging workflow rather than a single score. It deserves a serious referee; I'd send it out, and I'd ask the authors to add at least one controlled experiment with known injected flaws and a head-to-head with conventional metrics. With that validation, the paper would be stronger.","headline":"A genuinely useful diagnostic toolkit for synthetic tabular data, but the headline claim that XAI reveals generator flaws rather than detector artifacts is asserted, not yet demonstrated.","tokens_in":17975,"tokens_out":2515,"would_cite":true,"duration_ms":27240,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Explainable AI can show exactly why synthetic tabular data look wrong.","keywords":["synthetic data quality","generative artificial intelligence","explainable artificial intelligence","interpretable machine learning","synthetic data detection","tabular data","Shapley values","counterfactual explanations"],"falsifier":"Run a controlled experiment with an injected flaw: take a real tabular dataset, fit a generator, then deliberately break one known property—for example, randomizing one feature's values or severing one documented dependency—in a new synthetic sample. If the detection-plus-XAI pipeline does not rank that injected flaw at the top of feature importance and does not show a clear anomaly in the corresponding effect plot or Shapley decomposition, the claim that these tools reveal generator weaknesses is refuted.","tokens_in":17088,"feed_emoji":"🔍","tokens_out":9231,"duration_ms":83980,"temperature":0.7,"pith_summary":"Synthetic tabular data are usually graded with aggregate scores—statistical distances, utility measures, and detection accuracy—that can disagree and never say where generation failed. This paper proposes training a binary classifier to distinguish real from synthetic rows and then using explainable-AI tools on that classifier: permutation feature importance, partial dependence and ICE plots, Shapley values, and counterfactual explanations. The result is a feature-level audit that localizes unrealistic values, broken feature dependencies, and under- or over-represented regions of the data distribution. Demonstrations on two tabular datasets show the approach surfacing weaknesses that conventional quality metrics overlook.","feed_headline":"Explainable AI pinpoints why synthetic tabular data are detectable","feed_subtitle":"Feature-level diagnostics—not just quality scores—show exactly where a generator's rows diverge from real data.","key_machinery":"The load-bearing object is the detection classifier, a binary model $C$ mapping each row to its estimated probability of being real, trained on labeled real and synthetic data. It converts the unsupervised problem of judging a generative model into a supervised classification problem that XAI methods can address. Four families of explanations carry the argument: permutation feature importance and global Shapley values identify the features and interactions that separate real from synthetic; partial dependence and individual conditional expectation plots locate value regions where the classifier confidently flags synthetic rows; marginal and conditional Shapley values decompose individual predictions, with the conditional version respecting feature dependencies; and counterfactual explanations compute minimal, plausible feature changes that flip a synthetic row's classification to real. The qualitative distinction between marginal ('true to the model') and conditional ('true to the data') explanations is what lets the authors attribute a detected flaw to the generator rather than to the classifier's independence assumptions.","core_discovery":"The central claim is that a well-trained real-versus-synthetic detection classifier is not just a quality score: it is a supervised proxy whose explanations reveal why synthetic data are distinguishable. On the paper's own terms, the classifier answers four questions: which features and dependencies were hardest for the generator, how the generator behaves in low- and high-density regions, which features make individual synthetic or real rows look wrong, and which minimal changes would make a detected synthetic row look real. The authors show that global tools (permutation feature importance, Shapley feature and interaction importance, partial dependence and ICE plots) expose weak spots such as non-integer values in an integer feature, rare classes that are under-represented, and interactions that the generator does not retain. Local Shapley analysis further shows that a synthetic row can fail because of an implausible combination of individually plausible values, for instance a very young person with a school-leaving age, and counterfactual edits show the small changes needed to make that row pass as real. Conventional metrics, they argue, would report only that the synthetic data differ, not where or why.","pith_inferences":["A natural extension is to train the detector on real data the generator never saw, turning the same diagnostics into a privacy or generalization audit; the paper mentions this as future work.","The diagnostic plots could serve as regression tests in generator development: after a change, a regenerate-and-compare step would check whether the flagged PDP dip or interaction importance disappears.","Implicit in the worked examples is a practical rule: when features are locally dependent, prefer conditional Shapley values over marginal ones, because marginal attributions can blame the wrong feature for an unrealistic combination.","The counterfactual output could be reused as a data-cleaning prescription for individual synthetic rows, not only as a diagnosis of the generator."],"forward_implications":["Feature-importance rankings become a targeted debugging list: a generator's weak spots are the features and interactions the detector leans on most.","Partial-dependence and ICE plots turn quality assessment into a density-coverage map, showing which real-data regions the synthetic data over- or under-represent.","Local Shapley decompositions identify the specific value combinations that make an individual synthetic row look implausible, even when each feature alone looks fine.","Counterfactual edits give an actionable repair: for a flagged synthetic row, the minimal changes that make it pass as real indicate what the generator got wrong.","The same workflow is transferable across generators and datasets, so a fixed audit pipeline can compare two synthesizers or monitor one as it is modified."],"supporting_citations":[{"why":"Supplies the classifier two-sample test idea that makes real-versus-synthetic detection a supervised evaluation of generative models.","marker":"[5]"},{"why":"Closest prior work, applying a detection model and feature importances to tabular synthetic data; this paper extends that idea to a full XAI toolbox.","marker":"[22]"},{"why":"Defines permutation feature importance, the loss-based method used to rank features that are hardest to reproduce.","marker":"[53]"},{"why":"Defines partial dependence plots, used to locate feature-value regions where synthetic data look unrealistic.","marker":"[51]"},{"why":"Defines individual conditional expectation curves, used with PDPs to expose interactions and density coverage.","marker":"[52]"},{"why":"Provides conditional Shapley values that respect feature dependencies, the local explanation method used on dependent tabular features.","marker":"[54]"},{"why":"Provides tree-based Shapley interaction values, used to identify which feature dependencies the generator fails to retain.","marker":"[50]"},{"why":"Supplies the Monte Carlo counterfactual method used to find minimal realistic changes that make a detected synthetic row pass as real.","marker":"[21]"},{"why":"The latent-space diffusion synthesizer whose synthetic adult data are the paper's main demonstration.","marker":"[43]"},{"why":"The GAN-based synthesizer whose synthetic nursery data provide the secondary demonstration.","marker":"[32]"}],"fun_headline_variants":["Explainable AI shows exactly why synthetic data looks fake","XAI uncovers the specific flaws in your synthetic tabular data","Beyond quality scores: XAI pinpoints generator weaknesses","See why synthetic data fails: XAI explains the gaps","XAI reveals the 'why' behind synthetic data detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole diagnostic chain rests on one premise: the detection classifier is a trustworthy proxy for synthetic data quality, so a low accuracy means either a weak classifier or genuinely good synthetic data and the explanations reflect generator flaws rather than classifier artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Explainable AI shows exactly why synthetic data looks fake","XAI uncovers the specific flaws in your synthetic tabular data","Beyond quality scores: XAI pinpoints generator weaknesses","See why synthetic data fails: XAI explains the gaps","XAI reveals the 'why' behind synthetic data detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000748,"raw_usage":{"total_tokens":3328,"prompt_tokens":939,"completion_tokens":2389,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2308}},"tokens_in":555,"tokens_out":2389,"duration_ms":15906,"temperature":1.0,"reasoning_tokens":2308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:21:48.118457+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled experiment with an injected flaw: take a real tabular dataset, fit a generator, then deliberately break one known property—for example, randomizing one feature's values or severing one documented dependency—in a new synthetic sample. If the detection-plus-XAI pipeline does not rank that injected flaw at the top of feature importance and does not show a clear anomaly in the corresponding effect plot or Shapley decomposition, the claim that these tools reveal generator weaknesses is refuted.","supporting_citations":[{"cited_title":"Tabular data generation: Can we fool XGBoost ? In NeurIPS 2022 First Table Representation Workshop, 2022","cited_arxiv_id":null,"evidence_quote":"Closest prior work, applying a detection model and feature importances to tabular synthetic data; this paper extends that idea to a full XAI toolbox."},{"cited_title":"All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously","cited_arxiv_id":null,"evidence_quote":"Defines permutation feature importance, the loss-based method used to rank features that are hardest to reproduce."},{"cited_title":"Greedy function approximation: a gradient boosting machine","cited_arxiv_id":null,"evidence_quote":"Defines partial dependence plots, used to locate feature-value regions where synthetic data look unrealistic."},{"cited_title":"Peeking inside the black box: Visualizing statisti- cal learning with plots of individual conditional expectation","cited_arxiv_id":null,"evidence_quote":"Defines individual conditional expectation curves, used with PDPs to expose interactions and density coverage."},{"cited_title":"Explaining individual predictions when features are dependent: More accurate approximations to Shapley values","cited_arxiv_id":null,"evidence_quote":"Provides conditional Shapley values that respect feature dependencies, the local explanation method used on dependent tabular features."},{"cited_title":"MCCE: Monte carlo sampling of valid and realistic counterfactual explanations for tabular data","cited_arxiv_id":null,"evidence_quote":"Supplies the Monte Carlo counterfactual method used to find minimal realistic changes that make a detected synthetic row pass as real."},{"cited_title":"Mixed-type tabular data synthesis with score-based diffusion in latent space","cited_arxiv_id":null,"evidence_quote":"The latent-space diffusion synthesizer whose synthetic adult data are the paper's main demonstration."},{"cited_title":"Modeling tabular data using conditional GAN","cited_arxiv_id":null,"evidence_quote":"The GAN-based synthesizer whose synthetic nursery data provide the secondary demonstration."}],"review_version":1}