{"id":"b104efa3-dcdd-49a8-9b8e-7710b077229f","arxiv_id":"2501.09591","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Two PCA-based metrics, eigenvalue difference and first-eigenvector angle, are proposed for dataset similarity and shown to aid synthetic data and feature selection evaluation.","lead":"The paper proposes two PCA-based metrics for measuring inter-dataset similarity: the difference in explained variance and the angle between the first principal components. It applies them to evaluate synthetic data quality and to rank feature selection methods without training any model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Δλ's normalization is undefined: Eq. (4.7) assumes eigenvalues sum to 1, but Section 3's covariance eigenvalues sum to total variance (or d under standardization), so the metric can exceed 1 and is preprocessing-dependent.","rationale":"The reader's weakest assumption is exactly the load-bearing gap I identify: Eq. (4.7)'s normalization factor is derived under an eigenvalue-sum-to-1 precondition that the metric definition never states. This is not a matter of disagreeing with current consensus; it is a correctness and reproducibility issue internal to the paper. The empirical demonstrations are plausible and the instability discussion in Section 4.3 is honest, but every reported Δλ value and the claimed unit-interval range presuppose a preprocessing and eigenvalue-normalization choice. Section 7's admission that mean-subtracted and standardized data give different results confirms the ambiguity is real, not hypothetical. The proposed analytical counterexample settles the point directly: with standardized data and p=d, Eq. (4.7) as written yields Δλ = d. I therefore do not move the reader's verdict; CONDITIONAL remains appropriate, with the condition being that the authors must specify the normalization in the metric definition and recompute or qualify the reported values accordingly.","tokens_in":28560,"tokens_out":9549,"duration_ms":102263,"concrete_test":"Analytical check: for any d≥2, instantiate two standardized datasets whose correlation matrices have eigenvalue spectra (d,0,...,0) and (1,1,...,1) (e.g., perfectly correlated variables vs. uncorrelated unit-variance variables). Substitute into Eq. (4.7) with p=d; the result is d, not ≤1, proving the unit-interval claim requires dividing eigenvalues by d (or another explicitly stated normalization) before applying d/(d+p−2). Additionally, recompute Table 2's Δλ column under mean-centered raw data vs. z-score standardization to document the resulting ranking shift.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (4.7) defines Δλ = d/(d+p−2) Σ|λ_i − λ'_i| without stating what the λ_i are beyond 'eigenvalues' of a PCA. The normalization bound in Eq. (4.8) assumes the eigenvalue spectra sum to 1: one spectrum is (1,0,...,0) and the other is (1/d,...,1/d). But Section 3 defines PCA on the sample covariance matrix S = (1/(n−1)) X̃ᵀX̃, whose eigenvalue sum is the total variance, an arbitrary scale-dependent quantity. If the data are only mean-centered, as in Section 3, the sum is not 1; if the data are z-score standardized, S becomes a correlation matrix whose eigenvalues sum to d. In the standardized case with p=d, the worst-case spectra are λ=(d,0,...,0) and λ'=(1,...,1), giving Σ|λ_i−λ'_i| = 2d−2 and hence Δλ = d, not ≤1. Section 7 concedes that switching between mean-subtracted and standardized data changes the results. Therefore, as written, Eq. (4.7) is not a well-defined function on datasets: its claimed unit-interval range, the values in Tables 2 and 7, and any cross-dataset comparison all depend on an unstated normalization convention. This is the load-bearing weak point for the paper's primary quantitative claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two PCA-based metrics for inter-dataset similarity: Δλ, the normalized difference between eigenvalue spectra, and Δθ, the angle between first principal components. After reviewing PCA background, the authors analyze the metrics' normalization and invariance, then apply them to two tasks: evaluating synthetic data utility and evaluating feature selection results via the derived Average Angle Difference (AAD). The empirical sections report experiments on synthetic data from several generators (using SynthEval) and on 20 UCI datasets with four feature selection methods. The central claims are that the metrics are deterministic, model-agnostic, and capture holistic dataset structure, and that AAD can select the best feature selection result without a downstream model.","tokens_in":28908,"tokens_out":5702,"duration_ms":64396,"significance":"If the technical issues are resolved, the paper offers a simple, deterministic, and computationally light complement to model-dependent inter-dataset similarity measures. The authors provide an implementation on GitHub and evaluate on a substantial number of benchmark datasets, and the correlation analysis in Section 5.1 is a useful way to position the new metrics against existing utilities and privacy metrics. The proposed metrics have no fitted parameters (beyond the choice of p for Δλ), which is a genuine practical advantage. However, the paper's primary quantitative claims currently depend on unstated preprocessing and normalization choices, and the feature-selection application needs a sharper statement of what AAD actually measures.","major_comments":[{"comment":"The normalization of Δλ is not well-defined as stated. The derivation of the bound in Eq. (4.8) assumes that each eigenvalue spectrum sums to 1, with the 'pure noise' spectrum given by λ'_i = 1/d. But Section 3 defines PCA on the sample covariance matrix S = (1/(n−1)) X̃ᵀX̃, whose eigenvalues sum to the total variance; for mean-centered data this sum is an arbitrary scale-dependent quantity, and for z-score standardized data the correlation-matrix eigenvalues sum to d. Without an explicit convention (e.g., standardizing and dividing all eigenvalues by d), the claimed unit-interval range is not guaranteed: for standardized data with p = d, the worst-case spectra λ = (d,0,…,0) and λ' = (1,…,1) give Δλ = d, not a value in [0,1]. Section 7 concedes that switching between mean-subtracted and standard-deviation-normalized data changes the results. The definition in Eq. (4.7) must specify the preprocessing and the normalization of the λ_i, and the numerical values in Table 2 and the figures must be recomputed under that convention.","section":"Section 4.1, Eqs. (4.7) and (4.8)"},{"comment":"The proof that additive noise does not change the covariance matrix is only valid in expectation, not for finite samples. In Eq. (.2), the sample mean of the noise terms is claimed to be zero, but for a finite sample Σ_i δx_i is not exactly zero; similarly, the cross terms Σ_i x_i δy_i, Σ_i y_i δx_i, and the term η Σ_i δx_i δy_i in Eq. (.3) do not vanish exactly. The statement 'it has been shown that random noise does not change the elements of the covariance matrix' is therefore too strong. The privacy sanity-check claims in Section 5.3 rely on this result, so the appendix should either state the result as an expectation/probabilistic statement or provide a finite-sample bound.","section":"Appendix, Eqs. (.2) and (.3)"},{"comment":"The AAD criterion in Eq. (6.10) is partly mechanical: as more features are selected, the complement set F^C shrinks and fewer features are set to zero, so the average angle difference tends to decrease regardless of feature quality. Figure 6 confirms a clear downward trend as the number of selected features increases. Consequently, the claim that 'the minimum AAD value can successfully capture the best feature selection result' is not supported unless all compared feature selection outputs have the same cardinality, or unless the metric is adjusted for cardinality. The experimental section should state the selected-feature counts for each method, compare methods at equal cardinality, or otherwise demonstrate that AAD is not simply favoring larger feature subsets.","section":"Section 6.2 and 6.3, Eq. (6.10)"},{"comment":"The paper positions the metrics as providing a 'holistic perspective on the entire dataset', but Δθ in Eq. (4.9) uses only the first principal component of each dataset. While Δλ summarizes the full spectrum when p is chosen large enough, Δθ discards all principal components beyond the first, and in near-degenerate spectra the first eigenvector is not even uniquely defined. If the holistic claim is to be retained, the paper should either generalize Δθ to multiple principal components or explicitly qualify that Δθ measures only the dominant direction of variation.","section":"Abstract and Section 4.2"}],"minor_comments":[{"comment":"The text refers to 'Proofs and further details are supplied in Appendix 7', but the appendix has no numbered sections; the reference should simply say 'Appendix'.","section":"Section 5.3"},{"comment":"The definition of Δλ never states how the number of components p is chosen, even though both the sum and the normalization factor in Eq. (4.7) depend on p. The paper should discuss the choice of p and the sensitivity of results to it.","section":"Section 4.1"},{"comment":"The notation in the rotation invariance proof is inconsistent with Section 3: the projection is written as Z = XA = YQQᵀB = YB, but earlier Eq. (3.6) defines Z = X̃A. Clarify whether the data are centered and whether X in the appendix is already centered.","section":"Appendix, Eq. (.9)"},{"comment":"The 'noisy (10%)' row reports Δλ = 0.00 and Δθ = 0.00 exactly, which conflicts with the finite-sample noise invariance caveat raised in the appendix unless the values are rounded to two significant figures; this should be clarified.","section":"Table 2"},{"comment":"The labels 'pca_eigval' and 'pca_eigvec' are used without defining which proposed metric each corresponds to; the caption should state that these are Δλ and Δθ.","section":"Figure 4"},{"comment":"Reference [40] is a non-archival blog post; for a peer-reviewed venue it should be replaced by a citable source on PCA applications.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The normalization issue in Section 4.1 is the key technical obstacle: it is fixable by explicitly defining the preprocessing and eigenvalue normalization, but it affects the core definition and the reported numbers. The finite-sample noise invariance problem and the AAD cardinality issue are also load-bearing and need reworking. The authors should also decide whether to keep the 'holistic' language, since Δθ is explicitly a first-component measure. I see no evidence of citation or novelty problems beyond the minor reference issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful heuristic paper with a real gap between its theoretical claims and its definitions. The two metrics are simple and cheap, and the two applications show they track something meaningful. But Eq. (4.7) as written is not a well-defined function on datasets: the unit-interval bound in Eq. (4.8) only works if the eigenvalue spectra are normalized to sum to 1, and neither Section 3 nor the metric statement says to do that. Under standardization, eigenvalues sum to d, so Δλ can be d times larger than intended. The appendix's noise-invariance proof is expectation-only; finite-sample noise changes the covariance matrix, so the statement \"random noise does not change the elements\" is wrong as written. These are fixable, but they are the load-bearing parts of the \"fully deterministic\" claim.\n\nWhat is actually new: paired metrics Δλ and Δθ, a wrapper AAD for feature selection evaluation, and empirical evidence that they behave sensibly on real datasets. The correlation analysis with SynthEval metrics is a good sanity check; Fig. 3's tracking of TVAE training is a nice demonstration. The paper is honest in Section 7 about preprocessing dependence, which mitigates the normalization issue somewhat, but the definition should state the convention up front.\n\nSoft spots beyond the normalization: Δθ uses only the first PC, so \"holistic\" is oversold; the AAD decreases automatically as more features are selected because fewer features are zeroed, so comparing feature selection methods with different subset sizes needs care; the paper doesn't position against the principal-angle/subspace-similarity literature; and the GitHub link is not in the manuscript.\n\nWho it's for: people building or benchmarking tabular synthetic data generators, and people comparing feature selection methods without wanting to train classifiers. It's a heuristic toolkit, not a deep theoretical result. A serious referee should see it, but the normalization and the proof need fixing first.","headline":"Useful PCA-based heuristic for dataset similarity, but Eq. (4.7) needs an explicit normalization precondition and the noise-invariance proof is only expectation-level.","tokens_in":29336,"tokens_out":3492,"would_cite":false,"duration_ms":36514,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62H25"],"pacs":[],"model":"deepseek-v4-flash","headline":"Two PCA-based metrics, the difference in explained variance and the angle between first principal components, are claimed to give deterministic, model-independent measurements of inter-dataset similarity.","keywords":["inter-dataset similarity","principal component analysis","explained variance difference","angle difference","synthetic data evaluation","feature selection evaluation","model-agnostic metric","tabular data"],"falsifier":"Run the paper's synthetic-data benchmark twice on the same generated datasets, once with mean-subtracted data and once with data standardized by standard deviation; if the rankings or values of $\\Delta\\lambda$ and $\\Delta\\theta$ change, the metric's output depends on preprocessing choice, and the claimed normalization to $[0,1]$ is not intrinsic to the data.","tokens_in":28361,"feed_emoji":"📊","tokens_out":7951,"duration_ms":75796,"temperature":0.7,"pith_summary":"The paper sets out to measure how similar two numerical tabular datasets are by comparing their principal component structure instead of training any model. It proposes two metrics: the difference in explained variance ($\\Delta\\lambda$), a sum of absolute differences between eigenvalue spectra with a data-dimension-dependent normalization, and the angle difference ($\\Delta\\theta$) between the first principal components of the two datasets. Both metrics are deterministic for a given sample and need no labels, kernels, pretrained embeddings, or downstream tasks, which the paper argues distinguishes them from common similarity measures. The paper then argues that these metrics work in two practical roles: tracking and scoring synthetic-data quality during training, and, through an averaged angle difference called AAD, selecting the feature-selection result that best preserves the dataset's structure without running a classifier. A sympathetic reader would care because cheap, model-free dataset similarity would make synthetic-data auditing and feature-selection comparison more reproducible and less dependent on arbitrary modeling choices.","feed_headline":"Two PCA metrics size up dataset similarity without training a model","feed_subtitle":"Eigenvalue and first-component differences give deterministic checks for synthetic data quality and feature selection.","key_machinery":"The carrying object is the eigenvalue decomposition of the sample covariance matrix, specifically the ordered eigenvalue spectrum $\\lambda_1 \\ge \\cdots \\ge \\lambda_p$ and the first eigenvector $a_1$. The eigenvalues encode how much variance each principal direction explains, so $\\Delta\\lambda$ measures differences in how variance is distributed, while $a_1$ encodes the dominant direction of variation, so $\\Delta\\theta$ measures whether the two populations vary along different axes. The normalization in $\\Delta\\lambda$ is derived from a bound comparing a fully concentrated spectrum with a noise-like uniform spectrum, while the factor $2/\\pi$ in $\\Delta\\theta$ maps the minimum angle between the two first principal components onto the unit interval. AAD assembles these angle differences into a feature-selection criterion by setting each unselected feature to zero and averaging the resulting angle cost over all unselected features.","core_discovery":"The central discovery is that inter-dataset similarity for numerical tabular data can be read off the PCA decomposition of the two datasets: if two datasets share the same underlying statistics, their empirical covariance matrices and therefore their eigenvalue sequences and leading eigenvectors should agree, and any divergence is quantified by the two proposed metrics. The eigenvalue-based metric is $\\Delta\\lambda = \\frac{d}{d+p-2}\\sum_i |\\lambda_i - \\lambda_i'|$, where $d$ is the number of variables and $p$ the number of components considered, and it captures how the total variance is distributed across principal directions. The angle metric is $\\Delta\\theta = \\frac{2}{\\pi}\\min[\\arccos(a_1 \\cdot a_1'), \\arccos(a_1 \\cdot (-a_1'))]$, with the minimum taken because principal components are only defined up to sign, and it captures whether the dominant axes of variation point in the same directions. The paper further claims that averaging the angle differences contributed by zeroing out each unselected feature gives AAD, a model-agnostic score whose minimum across feature-selection methods agrees with the best classifier-based or clustering-based result in most of the benchmark datasets examined.","pith_inferences":["A natural extension the authors gesture at but do not develop is to use AAD as a stopping rule in forward feature selection, since the reported downward trend suggests a plateau once redundant features are added.","Because $\\Delta\\theta$ only reads the first principal component, two datasets with identical first eigenvectors but different later components could be rated very similar; a version averaging angles over several components would be a direct stress test of the method.","The preprocessing sensitivity noted in Section 7 implies that reproducible use of these metrics requires publishing a fixed standardization convention, since switching between mean-subtracted and standardized data changes the measured values.","If a consistent preprocessing convention is fixed, the same machinery applies beyond tabular data to any numerical representation, including images or text embeddings.",""],"forward_implications":["Synthetic-data quality can be monitored during training: the proposed metrics fall as a generative model improves and plateau when quality converges, offering a possible early-stopping signal.","Because the metrics are deterministic and independent of downstream tasks, evaluations using them avoid the variability introduced by classifier training, kernel choice, or embedding models.","Under standardization, the metrics are invariant to translation and scaling, and orthogonal rotations change only $\\Delta\\theta$ by the rotation angle, so they can sanity-check whether a supposedly synthetic dataset is merely an altered copy of real data.","AAD provides a model-agnostic way to rank feature-selection results: picking the result with the minimum AAD agrees with the best classifier-based or clustering-based result on most of the datasets tested.","AAD decreases as more features are selected, reflecting the marginal contribution of each feature to the dataset's principal structure.",""],"supporting_citations":[{"why":"Supplies the PCA formulation on which both metrics are built.","marker":"[26]"},{"why":"Supplies the mathematical properties of sample principal components, including eigenvalue ordering and the role of eigenvectors.","marker":"[12]"},{"why":"Provides the total-variation normalization precedent used to motivate the unit-interval factor for $\\Delta\\lambda$.","marker":"[5]"},{"why":"Supplies the kernel-based maximum mean discrepancy baseline that the proposed metrics are designed to avoid.","marker":"[6]"},{"why":"Supplies an embedding-based distance baseline that depends on a pretrained model, illustrating the model-dependence the paper argues against.","marker":"[8]"},{"why":"Supplies the Wasserstein-distance baseline, cited as computationally expensive and outlier-sensitive.","marker":"[25]"},{"why":"Establishes that synthetic-data evaluations often focus on marginal or pairwise statistics, motivating a holistic eigenvalue comparison.","marker":"[3]"},{"why":"Provides the benchmark suite used to compare the proposed metrics with utility and privacy metrics across many synthetic datasets.","marker":"[19]"},{"why":"Provides a recent dedicated feature-selection evaluation metric that still depends on a downstream model, serving as the contrast for AAD.","marker":"[30]"}],"fun_headline_variants":["Eigenvalue and angle metrics gauge dataset similarity","PCA eigenvalues and angles reveal how alike datasets are","PCA metrics read dataset similarity without fitting models","Eigenvalue difference and angle gap spot dataset mismatch","Metric pair reads dataset similarity from PCA decomposition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The unit-interval guarantee for $\\Delta\\lambda$ holds only when the eigenvalue spectra of both datasets are normalized to sum to one, and the paper never states this precondition in the metric definition; without a fixed standardization convention, the same pair of datasets yields different similarity values.","fun_headline_variants_meta":{"raw":{"variants":["Eigenvalue and angle metrics gauge dataset similarity","PCA eigenvalues and angles reveal how alike datasets are","PCA metrics read dataset similarity without fitting models","Eigenvalue difference and angle gap spot dataset mismatch","Metric pair reads dataset similarity from PCA decomposition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00086,"raw_usage":{"total_tokens":3708,"prompt_tokens":898,"completion_tokens":2810,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2740}},"tokens_in":514,"tokens_out":2810,"duration_ms":19409,"temperature":1.0,"reasoning_tokens":2740,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:51:29.609255+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's synthetic-data benchmark twice on the same generated datasets, once with mean-subtracted data and once with data standardized by standard deviation; if the rankings or values of $\\Delta\\lambda$ and $\\Delta\\theta$ change, the metric's output depends on preprocessing choice, and the claimed normalization to $[0,1]$ is not intrinsic to the data.","supporting_citations":[{"cited_title":"LIII. On lines and planes of closest fit to systems of points in space","cited_arxiv_id":null,"evidence_quote":"Supplies the PCA formulation on which both metrics are built."},{"cited_title":"Mathematical and Statistical Properties of Sample Principal Components","cited_arxiv_id":null,"evidence_quote":"Supplies the mathematical properties of sample principal components, including eigenvalue ordering and the role of eigenvectors."},{"cited_title":"A kernel two-sample test","cited_arxiv_id":null,"evidence_quote":"Supplies the kernel-based maximum mean discrepancy baseline that the proposed metrics are designed to avoid."},{"cited_title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","cited_arxiv_id":null,"evidence_quote":"Supplies an embedding-based distance baseline that depends on a pretrained model, illustrating the model-dependence the paper argues against."},{"cited_title":"Statistical aspects of Wasserstein distances","cited_arxiv_id":null,"evidence_quote":"Supplies the Wasserstein-distance baseline, cited as computationally expensive and outlier-sensitive."},{"cited_title":"A Dynamic Eval- uation Metric for Feature Selection","cited_arxiv_id":null,"evidence_quote":"Provides a recent dedicated feature-selection evaluation metric that still depends on a downstream model, serving as the contrast for AAD."}],"review_version":1}