{"id":"fec2767c-83f8-48f8-a99b-4c80cb0526f1","arxiv_id":"2504.20900","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper adapts FID and Inception Score to tabular data as FAED, FPCAD, and RFIS, and reports that FAED is the most sensitive to injected noise, mode drop, and mode collapse across three intrusion detection datasets.","lead":"This paper proposes three new metrics, FAED, FPCAD, and RFIS, for judging whether synthetic tabular data resembles real data, and tests them by injecting noise, removing common modes, and forcing value collapse into three network intrusion datasets. It matters because tabular synthetic data evaluation has no agreed standard, and the authors claim FAED detects failures that existing metrics miss.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim that FAED evaluates generative models is unsupported: experiments inject corruptions into real data but never train or sample a generative model, so sensitivity to hand-crafted perturbations need not transfer to real generative failures.","rationale":"The reader's weakest assumption identifies the same core concern: Section IV-B/IV-C assumes that injecting Gaussian noise, dropping common value combinations, and replacing rows with mean/mode values into real test data faithfully reproduces the failure modes of actual generative models. The paper never trains a GAN, VAE, or diffusion model and never evaluates the metrics on generated samples. My stress-test finds this concern to be load-bearing: without a real-model validation experiment, the abstract's claim that FAED 'effectively captures generative modeling issues' is unsupported, because sensitivity to hand-crafted corruptions does not imply sensitivity to the distributional shifts produced by actual generative models. The concern is not about disagreement with consensus; it is an internal validity gap between the experimental protocol and the claimed conclusion. Secondary issues, such as the undefined 'Synthesis Score' and the malformed code URL, further reduce confidence but are not the decisive problem. The contribution could be salvaged by re-framing the experiments as a sensitivity analysis of distribution distances under controlled corruption, or by rerunning the evaluation with actual generative models and deliberate model-level degradation. Since the reader already recommended REJECT and my analysis supports that recommendation, the verdict remains unchanged.","tokens_in":12310,"tokens_out":4500,"duration_ms":51528,"concrete_test":"Train at least one tabular generative model from each family (e.g., CTGAN, TVAE, and TabDDPM) on the same UNSW-NB15 train split used in the paper. Generate synthetic samples from each model and compute FAED between those samples and the real test set. Then create deliberately degraded variants of each generator by (a) early stopping, (b) training on only one class to force mode collapse, and (c) truncating the latent sampling range. Check whether FAED ranks the healthy generator above all degraded variants and whether FAED scores increase monotonically with the degree of degradation. If FAED fails to separate known-good from known-bad generators, the perturbation experiments in Sections IV-B and IV-C do not support the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that FAED 'effectively captures generative modeling issues overlooked by existing metrics.' The evidence for this claim rests entirely on Section IV-B/IV-C, where Gaussian noise, dropped value combinations, and mean/mode row replacements are applied to real test data and then compared against the training set. No GAN, VAE, diffusion model, or any other generative model is trained, sampled, or evaluated. The scoregen in Equation 6 is computed by comparing distorted real data with undistorted real data, not by comparing generated samples with real samples. This creates a proxy assumption that is load-bearing: hand-crafted corruptions of real data must faithfully reproduce the failure modes of actual generative models. That assumption is not validated and is questionable. For example, global Gaussian noise added to every feature produces independent, unstructured perturbations, whereas mode collapse in a trained model typically appears as concentrated or repeated latent-space samples; replacing rows with mean/mode values is not what a partially collapsed generator outputs. A metric can be sensitive to these specific corruptions yet fail to detect real generative defects such as covariance shrinkage, conditional mode dropping, or adversarial covariate shifts. The paper's abstract and conclusion generalize from perturbation sensitivity to generative-model evaluation without a bridging experiment, and no formal verification or released reproducible code offsets this gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three new evaluation metrics for tabular generative models: FAED (Fr\\'echet distance on autoencoder latent representations), FPCAD (Fr\\'echet distance on PCA projections), and RFIS (an entropy-based score using a random forest classifier). The authors evaluate these metrics against SDV Fidelity, TSTR, and TRTS on three intrusion-detection datasets by injecting three types of controlled corruption into real test data: Gaussian noise (quality decrease), removal of common value combinations (mode drop), and replacement of rows with mean/mode values (mode collapse). They report that FAED is the most sensitive to all injected corruptions, while FPCAD and RFIS show partial or inconsistent sensitivity. The paper claims that FAED 'effectively captures generative modeling issues overlooked by existing metrics.'","tokens_in":12394,"tokens_out":6616,"duration_ms":68936,"significance":"If the central claim were established, FAED would be a practical and computationally light diagnostic for tabular generative models, filling a real gap in the literature. The paper is also useful in identifying limitations of accuracy-based utility metrics like TRTS and in adapting FID/IS-style ideas to mixed-type tabular data. The proposed metrics are clearly defined and the comparative benchmark across three datasets is systematic. However, the current experimental evidence is restricted to hand-crafted distortions of real data and does not involve any generative model, so the significance as demonstrated is limited to a perturbation-sensitivity study rather than a validated evaluation framework for generative models.","major_comments":[{"comment":"The experimental protocol never trains or samples a generative model. In Section IV-C, step 4 defines scoregen as the score obtained by comparing distorted real test data against the training set, not by comparing generated samples with real data. All 'generative modeling issues' in Section IV-B are hand-crafted corruptions of real rows (Gaussian noise, dropped value combinations, mean/mode replacement). Consequently, the results in Tables II and III demonstrate only that FAED, FPCAD, RFIS, and the baseline metrics respond to these specific perturbations; they do not establish that any metric can evaluate the output of a GAN, VAE, or diffusion model. Since the abstract and conclusion claim that FAED 'effectively captures generative modeling issues,' a bridging experiment with actual generative models is required.","section":"IV-B, IV-C, Eq. (6)"},{"comment":"The validation is self-referential for distance-based metrics. FAED and FPCAD are Fr\\'echet distances between distributional summaries; adding Gaussian noise or discarding rows changes the mean and covariance, so the distance necessarily increases. The paper does not compare FAED with a simple baseline such as the Fr\\'echet distance computed on raw features, maximum mean discrepancy, or even a mean/covariance distance without an autoencoder. Without such controls, the sensitivity shown in Tables II and III is an expected property of any distributional distance and does not demonstrate that the autoencoder latent space adds diagnostic value. A control experiment or theoretical argument is needed to support the claim that FAED captures structural issues beyond trivial distribution shift.","section":"IV-C, Eq. (6), Tables II-III"},{"comment":"The FPCAD procedure is not fully specified. Section III-B states that PCA reduces dimensionality to 'principal components that preserve the most significant variance' but does not state the number of components or a variance threshold, while Figure 1 says the first two principal components are extracted. These are contradictory instructions, and the ambiguity makes FPCAD irreproducible. The authors should fix the number of components, report sensitivity to that choice, and align the figure with the text.","section":"III-B, Figure 1"},{"comment":"All reported values are point estimates without error bars, multiple seeds, or significance tests. The protocol includes randomness in the autoencoder training, PCA, random forest training, noise sampling, and train/test splitting, so the claimed consistent trends (e.g., 'FAED exhibits strong sensitivity') may not be statistically reliable. The paper should report means and standard deviations over repeated runs and test whether score changes are significant relative to baseline noise.","section":"V, Tables II-III"}],"minor_comments":[{"comment":"In Section V-B, the FAED single mode drop values for UNSW-NB15 are reported as -9.223 and -20.067, but Table II lists -8.289 and -21.241; the text and table should be reconciled.","section":"V-B, Table II"},{"comment":"Equation (6) uses a single formula for all metrics, but some metrics are higher-is-better and others lower-is-better; the 'relative score' therefore means opposite things depending on the metric. The arrows in Tables II and III partially address this, but the text should define the sign convention explicitly.","section":"Eq. (6), Tables II-III"},{"comment":"RFIS's entropy contrast depends on class balance; in the highly imbalanced intrusion datasets, p(y) is far from uniform, so the interpretation of 'diversity' via marginal entropy is confounded with class prior. Discuss or adjust.","section":"III-C"},{"comment":"The Gaussian noise is described by standard deviation alpha=0.1..0.5, but no feature scaling or normalization is specified, so the noise level's meaning depends on each feature's scale. Report preprocessing and consider per-feature noise.","section":"IV-B"},{"comment":"Figure 2 is missing from the manuscript; the text refers to it but only a summary paragraph appears. Insert the figure or refer to a table.","section":"V, Figure 2"},{"comment":"The footnote 'Code available' points to https://github.com/madhajj/Thesis Ali, which appears to be a personal repository rather than a project-specific release with versioning, dependencies, and license. Provide a proper repository or state the commit hash.","section":"Footnote, Code availability"}],"recommendation":"major_revision","confidential_remarks":"The paper reads like a thesis chapter in its current form: the code link is suspicious, Figure 2 is missing, and the validation does not match the strength of the central claim. The most serious issue is the absence of any generative model in the experiments; this is fixable within the scope of the paper, but it requires substantial new experiments with representative tabular generative models and appropriate baselines. I would not consider acceptance until that bridging evidence is provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper takes a sensible first step: adapt FID and Inception Score to tabular data, then sanity-check the adaptions with controlled perturbations. FAED, FPCAD, and RFIS are clearly defined (with caveats), and the idea of stress-testing metrics with noise, mode drop, and mode collapse is legitimate. Running this across three intrusion detection datasets and comparing against SDV Fidelity, TSTR, and TRTS is a reasonable amount of work. The raw sensitivity results are plausible: FAED does respond strongly to the injected distortions, and that is worth knowing.\n\nThe soft spot is the one the reader flagged, and it is load-bearing. The experiments never train a GAN, VAE, or diffusion model. The perturbations are hand-crafted corruptions of real test data, not outputs from a generative model. A metric can be sensitive to added Gaussian noise or mean-value replacement yet still rank real generative models incorrectly. The paper's own framing, claiming that FAED detects \"generative modeling issues,\" is not supported by the design. The proxy assumption is unvalidated, and the paper offers no bridging experiment.\n\nThere are additional, smaller problems. No error bars or significance tests appear anywhere; numbers like -1.381 vs -31.670 in a single run are not convincing. The FPCAD description does not specify whether PCA is fit jointly on real and synthetic data. If it is fit separately, the Fréchet distance is computed in different coordinate systems, which is invalid. RFIS has an internal inconsistency: Section III-C says higher is better, but the noise experiments show RFIS increasing with corruption and interpret that as degradation. The tables' arrow conventions only make this worse. \"Synthesis Score\" is mentioned in the summaries but never defined. The code URL on line ~160 is malformed (contains a space). Autoencoder and random forest hyperparameters are not reported.\n\nThe citation pattern is fine; the self-citations are on-topic and appropriate. The duplicated Esteban reference is sloppy but minor.\n\nWho is this for? People actively building tabular generative models or evaluation toolkits for them. They would get a useful starting point, but they should not trust the headline claim. The paper deserves a serious referee, not a desk reject, but the referee should ask for a major revision: train at least one real generative model (say CTGAN or TabDDPM), show whether the metrics rank models correctly, add error bars, clarify FPCAD, and fix the RFIS direction. If the authors can do that, the contribution could be solid.","headline":"Three FID-style metrics for tabular data, but the experiments never train a generative model, so the central claim outruns the evidence.","tokens_in":13053,"tokens_out":2870,"would_cite":false,"duration_ms":33137,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that FAED, a Fréchet distance computed on autoencoder latents, detects quality decrease, mode drop, and mode collapse in tabular data where existing metrics fail.","keywords":["tabular data","generative model evaluation","Fréchet distance","autoencoder","mode collapse","mode drop","quality decrease","intrusion detection"],"falsifier":"Train a generative model on one of the three datasets with a known, controlled weakness—for example, a GAN that produces only one class, or a diffusion model trained on a subset of modes—and compute FAED between real and generated rows. If FAED's relative score stays near the baseline or fails to rank the weakened model below a healthy one, the central claim that FAED detects real generative mode collapse is refuted.","tokens_in":11937,"feed_emoji":"📊","tokens_out":8869,"duration_ms":82044,"temperature":0.7,"pith_summary":"This paper tries to establish that tabular-data generative models need their own evaluation metrics, and that three new metrics—FAED, FPCAD, and RFIS—meet that need better than existing scores. The central claim is that FAED, a Fréchet distance computed on the latent representations of a pre-trained autoencoder, reliably detects all three failure modes the authors inject into real data: noise-driven quality decrease, mode drop, and mode collapse. FPCAD, a PCA-based variant, detects some but behaves inconsistently across datasets; RFIS, an entropy-based score, works for noise and collapse but not mode drop. The support comes from perturbing three intrusion-detection datasets (UNSW-NB15, TON-IoT, CICIDS-2017), not from training generative models and scoring their outputs, so the paper's case rests on whether those perturbations faithfully stand in for real generative failure. If the claim holds, FAED would give practitioners a single quantitative signal for ranking tabular generators where current metrics give conflicting or flat answers.","feed_headline":"Latent-distance score catches hidden flaws in synthetic tables","feed_subtitle":"A Fréchet distance on autoencoder latents flags noise, missing modes, and collapsed modes that older metrics overlook.","key_machinery":"The central object is the Fréchet distance (Wasserstein-2 distance) between two multivariate Gaussians, Equation (1): $\\lVert\\mu_r - \\mu_s\\rVert^2 + \\mathrm{Tr}(\\Sigma_r + \\Sigma_s - 2(\\Sigma_r\\Sigma_s)^{1/2})$, applied to feature vectors extracted by a pre-trained autoencoder for FAED or by PCA for FPCAD. RFIS replaces this with an entropy gap, $\\mathbb{E}[H(p(y)) - H(p(y|x))]$, between the marginal and conditional label distributions from a random forest. The Fréchet terms do the detection work: they compare the location and spread of latent representations, so any perturbation that shifts the latent mean or inflates the covariance registers as a larger distance.","core_discovery":"On the paper's own terms, the discovery is that a distance computed in autoencoder latent space separates intact tabular data from data degraded in ways that mimic generator failures. FAED computes Equation (1), the Fréchet distance between the mean and covariance of latent vectors from real rows and from a candidate generator, so a lower value means a closer match. In the three intrusion-detection datasets, the relative FAED score moves sharply and monotonically with increasing noise, with successive removal of the most common value combinations, and with replacement of rows by mean/mode prototypes, while SDV Fidelity, TRTS, and TSTR stay flat or erratic. The paper concludes that FAED captures structural distortion that existing metrics overlook and proposes it as a general-purpose evaluation metric for tabular generative models; FPCAD and RFIS are presented as partial alternatives with known limitations.","pith_inferences":["Because FAED needs only a pre-trained autoencoder and a distance computation, it could be embedded in automated model-selection loops for tabular generators; the paper does not study that use, but the metric's cost is concentrated in the one-time autoencoder training.","FAED and FPCAD both assume Gaussian latent distributions, so their disagreement on datasets like CICIDS-2017 is a testable probe for when that approximation breaks; adding a non-parametric distance as a control would show whether the assumption is the culprit.","The injection protocol itself—noise, mode drop, mode collapse—could be reused as a standardized stress test for any future tabular evaluation metric, independent of whether FAED is adopted."],"forward_implications":["FAED can serve as a single quantitative score for ranking tabular generative models, because it is the only tested metric that responds to all three injected failure modes.","Relying on SDV Fidelity or TRTS alone is risky: these scores can stay flat while the data has lost structure.","FPCAD provides a pre-training-free option where training an autoencoder is impractical, but its scores must be read dataset-by-dataset.","RFIS complements the Fréchet metrics for quality and collapse detection, but should not be trusted for mode drop."],"supporting_citations":[{"why":"It supplies the Fréchet Inception Distance formulation that FAED and FPCAD adapt to tabular latent spaces.","marker":"[2]"},{"why":"It supplies the Inception Score entropy logic that RFIS adapts to tabular labels.","marker":"[1]"},{"why":"It provides the autoencoder architecture whose latent layer FAED uses to represent rows.","marker":"[24]"},{"why":"It defines SDV Fidelity, one of the existing metrics the paper benchmarks against.","marker":"[26]"},{"why":"It defines the TSTR and TRTS training-and-testing baselines used in the comparison.","marker":"[3]"}],"fun_headline_variants":["Autoencoder distance exposes synthetic-table flaws old metrics miss","Latent distance score flags synthetic table flaws old metrics miss","New metric for synthetic tables beats old fidelity and utility scores","Frechet distance on latents catches hidden defects in synthetic data","New table metric finds flaws old ones ignore"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that adding noise, dropping the most common value combinations, and replacing rows with mean and mode prototypes faithfully reproduces the ways real generative models fail; the paper never trains a GAN, VAE, or diffusion model to check this.","fun_headline_variants_meta":{"raw":{"variants":["Autoencoder distance exposes synthetic-table flaws old metrics miss","Latent distance score flags synthetic table flaws old metrics miss","New metric for synthetic tables beats old fidelity and utility scores","Frechet distance on latents catches hidden defects in synthetic data","New table metric finds flaws old ones ignore"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001484,"raw_usage":{"total_tokens":5922,"prompt_tokens":869,"completion_tokens":5053,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":4975}},"tokens_in":485,"tokens_out":5053,"duration_ms":35276,"temperature":1.0,"reasoning_tokens":4975,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:16:23.809300+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a generative model on one of the three datasets with a known, controlled weakness—for example, a GAN that produces only one class, or a diffusion model trained on a subset of modes—and compute FAED between real and generated rows. If FAED's relative score stays near the baseline or fails to rank the weakened model below a healthy one, the central claim that FAED detects real generative mode collapse is refuted.","supporting_citations":[{"cited_title":"Improved techniques for training GANs,","cited_arxiv_id":null,"evidence_quote":"It supplies the Inception Score entropy logic that RFIS adapts to tabular labels."}],"review_version":1}