{"id":"7265f1da-7080-4e46-93e8-e319e8b88616","arxiv_id":"2502.08856","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Two new evaluation metrics, a network-graph similarity score and a holdout-based privacy ratio, reveal that current tabular generative models fall short on NYC taxi trip data.","lead":"This paper benchmarks six tabular data generators on New York City taxi data and finds they struggle to capture the structure of transportation networks. It introduces two new evaluation metrics, a graph-similarity score and a privacy-leakage ratio, and uses them to argue that domain-specific generative models are needed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Graph-metric result depends on unexplained alignment between 2015 lat/lon training data and March 2019 zone-based data; the claimed 'significant gap' cannot be interpreted without specifying how synthetic zone trips were obtained.","rationale":"The reader's weakest assumption pinpoints exactly the same concern: the graph metric's dataset alignment is unexplained. This is the most load-bearing issue because the abstract's second headline claim is specifically about the graph metric. The paper's other evidence, such as downstream task scores and coverage, supports the first part of the central claim (models often fail to perform consistently), but the 'significant gap' claim stands or falls with Table 3. The manuscript itself flags the issue only obliquely: Section 5.1 says the March 2019 zone-based data is used to satisfy the graph metric's zone requirement, and the Table 3 note says 'here we use the dataset with zone-based locations.' Neither statement reveals whether the generative models were retrained on that dataset or how synthetic zone trips were derived. Without that information, the comparison G tr syn / G te syn could be an artifact of mapping, schema mismatch, or a non-comparable training setup. This is an internal incompleteness rather than a disagreement with consensus, and it is fixable, so the conditional verdict remains appropriate. No change to the reader's verdict is needed, but the condition should explicitly require the authors to specify and justify the dataset alignment for the graph metric.","tokens_in":10796,"tokens_out":4017,"duration_ms":39232,"concrete_test":"Rerun Table 3 under two explicitly specified pipelines and report both. Pipeline A: train all six models on the same March 2019 zone-level dataset with the same 40,000/20,000 split, treating zone as a categorical variable. Pipeline B: train on the 2015 lat/lon data, then map both real and synthetic pickup/dropoff coordinates to TLC zones using the official zone shapefile, and compute SG on the resulting zone-pair counts. If the SG values from the two pipelines differ materially, or if the model ordering changes (for example, TabDDPM is no longer the worst), then the reported 'significant gap' depends on the currently unspecified dataset alignment rather than on a robust property of the models.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline 'significant gap' rests on Table 3, which reports the Section 4.1 graph metric over zone-to-zone trips. Section 5.1 introduces two different datasets: the 2015 Green Taxi data with precise longitude/latitude coordinates, used for all other experiments, and a March 2019 Green Taxi dataset where locations are zones rather than coordinates. The text never states which dataset the generative models were trained on for Table 3, nor how synthetic zone trips were obtained from models trained on the 2015 data. The only clue is the result note 'here we use the dataset with zone-based locations,' which is insufficient. If models were trained on 2015 lat/lon data and synthetic coordinates were later mapped to zones, the graph metric conflates model quality with geocoding or mapping artifacts. If models were retrained on the March 2019 zone dataset, the paper omits the training setup and Table 3 becomes incomparable with the other metrics. In either case, the central assertion that the novel graph metric reveals a significant gap between synthetic and real data is not supported until the alignment is specified. A secondary issue is that 'significant' is used without a statistical test, but the dataset-alignment gap is the primary load-bearing problem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a systematic comparison of six tabular generative models (Gaussian Copula, CTGAN, TVAE, CTABGAN, STaSy, TabDDPM) on New York City Green Taxi data. It evaluates downstream-task R², Wasserstein distance, coverage, DCR-based privacy, and two proposed metrics: a graph similarity metric (Section 4.1, Eq. 10) comparing the total-variation distance between real and synthetic edge-fraction distributions over zone pairs, and a Distance-to-Closest-Record ratio (rDCR, Section 4.2) that compares training-data and holdout-data distances to synthetic records. The results indicate TabDDPM has the best downstream and coverage performance, CTABGAN and TabDDPM have the smallest Wasserstein distances, all models show a large graph-similarity gap relative to the real-real reference (73.17), and Gaussian Copula has rDCR<1 at small percentiles. The abstract concludes that existing tabular models do not perform as consistently as claimed and that the graph metric reveals a significant gap between synthetic and real transportation networks.","tokens_in":11123,"tokens_out":5156,"duration_ms":48531,"significance":"The study addresses a timely question and brings two useful diagnostic ideas: a zone-graph structural similarity and a holdout-based distance ratio for privacy. The evaluation is unusually broad (six models, multiple utility and privacy axes) and the code is promised. If the dataset-alignment issue in Table 3 is resolved, the graph metric could be a valuable addition to the tabular-synthesis evaluation toolbox. At present, however, the paper's headline 'significant gap' claim is not interpretable without knowing how synthetic zone trips were generated, and the privacy-leakage conclusion is a proxy-based inference rather than a demonstrated attack. The manuscript is therefore promising but requires a substantial revision.","major_comments":[{"comment":"The graph metric experiments use a different dataset than all other experiments, but the paper never specifies how the generative models were trained for Table 3. Section 5.1 introduces the March 2019 Green Taxi zone-based dataset only for the graph metric, and Section 5.4 says only 'here we use the dataset with zone-based locations.' If models are trained on 2015 lat/lon data and synthetic coordinates are mapped to zones, the graph metric confounds model quality with a coordinate-to-zone mapping; if the models are retrained on March 2019 zone data, the training setup is omitted and Table 3 is not comparable with Tables 1, 2, 4, and 5. This is load-bearing because the abstract's 'significant gap' conclusion rests on Table 3. Please state which dataset was used for training and sampling, describe the mapping/retraining procedure, and if the graph metric requires a separate dataset, report the corresponding experimental setup.","section":"5.1, 5.4 and Table 3"},{"comment":"The statement that Gaussian Copula 'is vulnerable to membership inference attacks at very small values of α' is not supported by an actual attack. The paper defines vulnerability as rDCR<1 in Section 4.2, so the observed rDCR<1 at small percentiles is a direct consequence of the metric's definition, not evidence of a successful membership inference attack. A distance-ratio proxy can be a useful diagnostic, but the claim should be either reframed as a distance-based proxy finding or supplemented by running a standard membership inference attack and relating rDCR to attack success.","section":"4.2, 5.4, Table 5, Figure 1"},{"comment":"All model rankings are based on point estimates without significance tests, and several pairwise comparisons have overlapping standard deviations (e.g., Table 1 dwn syn tr: GaussianCopula 77.55±1.20 vs CTGAN 78.05±2.29). Claims such as 'TabDDPM achieves the best downstream task performance' and 'CTABGAN and TabDDPM have the best performance' in Section 5.4 are therefore stronger than the reported statistics justify. Please provide significance tests or explicitly state that the rankings are descriptive and not statistically verified.","section":"5.4, Tables 1-4"}],"minor_comments":[{"comment":"Clarify how dα is computed for the training and holdout sets, including whether the holdout set has the same size as the training set and how ties near the percentile boundary are handled.","section":"4.2"},{"comment":"Add a note explaining why STaSy entries are N/A and why G tr te is constant across models; the current text mentions these issues only in the body.","section":"Table 3 caption"},{"comment":"State whether the 40,000 training rows are a random sample from the 19.2 million rows, whether any stratification or random seed was used, and whether the March 2019 zone dataset was sampled in the same way.","section":"5.1"},{"comment":"Fix the unmatched parenthesis in the definition of Π(Pr, Pg) in the Wasserstein distance formula.","section":"Eq. (9)"},{"comment":"Make the column headers consistent with the explanations in the text (e.g., 'dwntr tr' vs 'dwn tr tr') and mark the reference-only columns explicitly in the table rather than only in the prose.","section":"Table 1"},{"comment":"Report training time and sampling time separately for the diffusion-based models, since the current combined running time mixes two different stages and makes cross-model complexity comparisons difficult to interpret.","section":"Figure 2"},{"comment":"The claim that TabDDPM 'suffers severe mode collapse' is an inference from a low graph score; a direct diagnostic such as the number of unique generated zones or a cluster-based mode-collapse measure would strengthen this conclusion.","section":"5.4 (Graph Similarity Metric)"}],"recommendation":"major_revision","confidential_remarks":"The dataset-alignment ambiguity in Table 3 is the main risk to the paper's central claim. The GitHub repository was not verified during review, so the authors should be asked to confirm that the released code reproduces Table 3 exactly and to document the zone-generation pipeline. The paper is within scope as an empirical evaluation study, but the abstract's 'significant gap' wording should be calibrated to the actual experimental support after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a useful benchmark: it runs six tabular generators on NYC taxi data and measures downstream utility, Wasserstein distance, coverage, and two new diagnostics. The graph similarity score is TV distance between zone-to-zone trip distributions—simple, but well matched to the structure of transportation data. The rDCR metric compares percentile distances from training and holdout data to synthetic data; it is a clear improvement over the plain DCR used in earlier work, and the percentile tail is the right place to look for membership inference risk. The experimental setup is honest: three model runs, fifteen sampling iterations, means and standard deviations, and they report failures like STaSy OOM and TabDDPM mode collapse instead of hiding them. Code is linked.\n\nThe main problem is the dataset alignment behind the graph metric, which carries the abstract's \"significant gap\" claim. Section 5.1 describes two datasets: 2015 Green Taxi with lat/lon coordinates for all the utility/similarity/privacy experiments, and March 2019 Green Taxi with zone-based locations for the graph metric. The paper never says which dataset the six models were trained on for Table 3, or how synthetic zone trips were produced. If models trained on 2015 coordinates were mapped into 2019 zones, the graph metric conflates model error with mapping artifacts; if they retrained on 2019 zone data, the setup is missing and Table 3 is incomparable with the other metrics. Either way the headline result is not interpretable until this is specified. That is fixable with a paragraph, but it is load-bearing right now.\n\nSmaller issues: \"significant gap\" is used without a statistical test—reasonable as shorthand, but the word carries weight. The rDCR finding that Gaussian Copula is vulnerable at small percentiles follows directly from defining vulnerability as rDCR < 1; that is legitimate as a diagnostic, but the paper should not present it as an actual membership inference attack. Running one real distance-based attack would validate the claim. Model rankings also lack significance tests; with fifteen sampling iterations a paired test is cheap.\n\nThese are addressable. The benchmark itself is useful, and the transport-specific graph metric is a reasonable idea that others will build on. The paper deserves a serious referee; I would ask for the dataset alignment to be spelled out and the privacy claim tempered or validated before acceptance.","headline":"Useful transportation-data benchmark with two sensible new diagnostics, but the headline graph-metric result is currently uninterpretable because the paper never explains how the zone-based synthetic trips were produced.","tokens_in":11577,"tokens_out":2124,"would_cite":false,"duration_ms":21383,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generic tabular generative models, tested on New York City taxi data, do not transfer to transportation data: synthetic trip networks are structurally far from real ones, and standard privacy metrics miss the models that actually leak.","keywords":["synthetic tabular data","transportation networks","graph similarity metric","NYC taxi data","privacy leakage","membership inference","generative model evaluation","mode collapse"],"falsifier":"Retrain the same six models on the March 2019 zone-level data—where the zone column is the target—and recompute $S_G$ against real zone graphs. If the scores approach the real train/test reference value 73.17, the reported gap was an artifact of mismatched training and evaluation data rather than a general property of the models.","tokens_in":10632,"feed_emoji":"🚕","tokens_out":5953,"duration_ms":54697,"temperature":0.7,"pith_summary":"This paper tests six off-the-shelf tabular generative models—Gaussian Copula, CTGAN, TVAE, CTABGAN, STaSy, and TabDDPM—on New York City green taxi trip data. The authors want to show that models developed for generic spreadsheets do not transfer to transportation data, where trips collectively form a network. They introduce a graph metric that compares real and synthetic trip networks and an improved privacy metric based on the ratio of training-data to holdout-data distances to the nearest synthetic record. Their results find a large structural gap between synthetic and real networks across all models, plus widespread mode collapse and one privacy vulnerability that standard metrics miss.","feed_headline":"New graph metric exposes gap in synthetic taxi data","feed_subtitle":"Six tabular generators were tested on NYC taxi trips; none preserved the city's trip network structure.","key_machinery":"The load-bearing object is the graph similarity metric $S_G$, computed on zone-level graphs whose nodes are NYC taxi zones and whose edges are trips between zones, weighted by the fraction of all trips. $S_G$ subtracts the total variation distance between the real and synthetic edge-fraction distributions from 1, so it measures how well synthetic trips preserve the collective network. The second mechanism is the Distance to Closest Record ratio ($rDCR$), the ratio of the $\\alpha$-percentile distance from training data to the nearest synthetic record over the same distance for holdout data; a ratio below 1 flags overfitting and membership-inference risk.","core_discovery":"On the paper's own terms, the central discovery is that the performance of tabular generative models is not portable to transportation data: rankings change with the metric, and no model reproduces the zone-to-zone network structure of real taxi trips. The graph metric scored every model far below the reference similarity between two real samples from the same dataset, and TabDDPM—the best overall on downstream prediction and coverage—scored worst on network structure. The proposed privacy ratio also overturns the ranking that the standard nearest-neighbor distance gives: Gaussian Copula looks safest by raw distance yet is the only model vulnerable to distance-based membership inference at small percentiles.","pith_inferences":["If the graph metric is equally sensitive on other mobility datasets, it could become a standard diagnostic for any synthetic data meant to preserve network flow, not just taxi trips.","A direct retraining experiment on the zone-level dataset would tell whether the low graph scores come from the models themselves or from feeding them coordinates and evaluating on zones.","The rDCR idea could generalize to other record-level data where overfitting is the main privacy risk, such as medical or financial tables."],"forward_implications":["If a transport agency publishes synthetic taxi or ride-hailing data from a generic tabular generator, downstream users should not assume the synthetic trip network reflects real flows.","Model rankings depend on the evaluation lens: TabDDPM dominates downstream prediction and coverage, while network-structure fidelity is poor for all six models.","A privacy report based only on raw distance to the closest real record can be misleading; the percentile-based ratio should accompany it.","The zone column with hundreds of categories is a specific failure point, suggesting categorical cardinality is a key challenge for tabular generators."],"supporting_citations":[{"why":"Supplies CTGAN and TVAE, two of the six compared models, and the conditional sampling design the evaluation tests.","marker":"[30]"},{"why":"Supplies CTABGAN and the raw DCR privacy metric that the paper's rDCR is designed to improve.","marker":"[31]"},{"why":"Supplies TabDDPM, the diffusion baseline that ranks best overall but scores lowest on graph similarity.","marker":"[16]"},{"why":"Supplies STaSy, the score-based diffusion baseline included in the comparison.","marker":"[14]"},{"why":"Supplies Gaussian Copula, the statistical baseline whose privacy risk is exposed by the new rDCR metric.","marker":"[22]"},{"why":"Defines membership inference attacks, the threat model behind the proposed rDCR privacy metric.","marker":"[24]"},{"why":"Supplies the coverage metric used to measure diversity and detect mode collapse.","marker":"[21]"}],"fun_headline_variants":["Taxi trip network structure lost in all synthetic models","Graph metric shows synthetic taxi data fails network test","Best synthetic taxi model still misses network structure","Privacy metric flips safety rankings for synthetic taxi data","Standard tabular generators fail on taxi network topology"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline gap result assumes the zone-level synthetic graphs used for scoring were produced in a way that is valid to compare with the real zone graph; the paper does not state whether the models were retrained on zone-based data or how zone trips were derived from models trained on coordinate data.","fun_headline_variants_meta":{"raw":{"variants":["Taxi trip network structure lost in all synthetic models","Graph metric shows synthetic taxi data fails network test","Best synthetic taxi model still misses network structure","Privacy metric flips safety rankings for synthetic taxi data","Standard tabular generators fail on taxi network topology"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000454,"raw_usage":{"total_tokens":2269,"prompt_tokens":920,"completion_tokens":1349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1277}},"tokens_in":536,"tokens_out":1349,"duration_ms":10551,"temperature":1.0,"reasoning_tokens":1277,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T23:27:01.230450+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same six models on the March 2019 zone-level data—where the zone column is the target—and recompute $S_G$ against real zone graphs. If the scores approach the real train/test reference value 73.17, the reported gap was an artifact of mismatched training and evaluation data rather than a general property of the models.","supporting_citations":[{"cited_title":"Advances in neural information processing systems 32 (2019) 2, 3, 4, 9","cited_arxiv_id":null,"evidence_quote":"Supplies CTGAN and TVAE, two of the six compared models, and the conditional sampling design the evaluation tests."},{"cited_title":"In: Asian Conference on Machine Learning","cited_arxiv_id":null,"evidence_quote":"Supplies CTABGAN and the raw DCR privacy metric that the paper's rDCR is designed to improve."},{"cited_title":"In: International Conference on Machine Learn- ing","cited_arxiv_id":null,"evidence_quote":"Supplies TabDDPM, the diffusion baseline that ranks best overall but scores lowest on graph similarity."},{"cited_title":"In: International Conference on Machine Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the coverage metric used to measure diversity and detect mode collapse."}],"review_version":1}