Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Measuring Time-Series Dataset Similarity using Wasserstein Distance

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the Wasserstein distance between multivariate normal models of two time-series datasets is a valid similarity measure and a strong empirical predictor of forecasting foundation-model inference loss.

desk verdict Useful distance for time-series datasets, but the performance-estimation claim rests on selected examples and needs full correlation reporting. read the letter →

arxiv 2507.22189 v1 pith:ANYNEV2L submitted 2025-07-29 cs.LG cs.AI

Hongjie Chen , Akshay Mehra , Josh Kimball , Ryan A. Rossi This is my paper · ORCID
classification cs.LGcs.AI
keywords time-seriesdatasetsimilarityWassersteindistancemultivariatenormaldistributionfoundationmodelinferencetransferlearningout-of-distributionevaluationforecasting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that a time-series dataset can be represented by a single multivariate normal distribution, fitted to fixed-length windows sampled from the dataset, and that the Wasserstein distance between two such distributions measures how similar the two datasets are. Treating datasets this way lets the authors compute a closed-form distance that combines differences in the mean time-series shape and in the covariance structure. The authors claim this distance is informative for practical forecasting decisions: across 30 real-world datasets, the Wasserstein distance between a source and a target dataset correlates strongly (>0.60) with the inference loss of two foundation models, in both out-of-distribution and transfer-learning settings. If the claim holds, researchers could estimate how well a pretrained model will do on a new dataset without running the model, and could pick finetuning source datasets by distance instead of trial and error.

What carries the argument

The load-bearing object is the closed-form 2-Wasserstein distance between two multivariate normal distributions, given by $d^{2}$ = ||mu_X - mu_Y||^2 + tr(Sigma_X + Sigma_Y - 2 $\sqrt$(Sigma_X Sigma_Y)). The first term captures differences in the average time-series shape, the second term captures differences in covariance structure, and the whole expression is a metric. This formula converts dataset comparison into a parameter-estimation step followed by a matrix computation, which is what makes the approach model-agnostic and much cheaper than finetuning.

What would settle it

Take a dataset whose windows are strongly multimodal or highly autocorrelated, compute the MVN-based Wasserstein distance from it to several other datasets, and compare the resulting ranking with actual inference losses. If the correlation with inference loss drops near zero or reverses on such datasets while staying high on near-Gaussian ones, the MVN assumption is the load-bearing part of the claim rather than the distance itself. A simpler check is to compute the distance from a dataset to a time-shuffled copy of itself; a faithful similarity measure should give near-zero distance to a shuffled copy if temporal order within windows is irrelevant.

Watch

Extended reading notes

Core claim

The central claim is that dataset similarity reduces to a distributional problem: after sampling N windows of length L from each dataset, estimate a multivariate normal for each dataset and use the 2-Wasserstein distance between the fitted normals as the dataset distance. The authors show that the resulting pairwise distance matrix separates 30 Gluonts datasets into clusters that match domain knowledge, and that the distance correlates with the inference loss of Lag-llama and PatchTST beyond 0.60 in out-of-distribution and transfer-learning settings. The paper also claims that the approach runs in O((N+L)$L^{2}$) time, making it scalable when the number of samples is much larger than the window length. The authors acknowledge that some datasets show negative correlations and attribute these outliers to deviations from the multivariate-normal assumption.

Load-bearing premise

The sampled L-length windows of each dataset are independent and identically distributed draws from one multivariate normal distribution; if a dataset's windows are not well described by a single Gaussian, the distance may be measuring a poor approximation rather than true similarity.

Editorial extensions

If this is right

  • Researchers can rank candidate source datasets for finetuning by Wasserstein distance alone, skipping expensive finetuning runs.
  • A foundation model's expected loss on an unseen target dataset can be estimated before running inference, enabling safer deployment decisions.
  • Dataset families with shared dynamics, such as electricity, traffic, solar, and Wikipedia page views, become discoverable through unsupervised clustering of the pairwise distance matrix.
  • The same distance can serve as an input to fairness checks by testing whether similar datasets receive similar model outputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the MVN representation discards within-window temporal order, the method implicitly treats time-series windows as exchangeable; a natural extension is to test whether a version that preserves temporal structure, for example via autocorrelation or spectral summaries, predicts inference loss even better on highly autocorrelated data.
  • The reported negative-correlation outliers suggest that the approach could be paired with a Gaussianity diagnostic that flags datasets for which the distance should not be trusted.
  • The same mean-plus-covariance Wasserstein formula may transfer to other data types, such as image patches or tabular slices, whenever datasets are represented as collections of fixed-length feature vectors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a distribution-based distance between time-series datasets. Each dataset is represented by a multivariate normal (MVN) distribution whose mean and covariance are estimated from N sampled windows of length L, and the dataset distance is the closed-form 2-Wasserstein distance between the two MVNs. The method is evaluated on 30 Gluonts datasets through pairwise distance heatmaps and graph visualization, compared with Euclidean, DTW, and clustering-based distances, and tested as a predictor of out-of-distribution and transfer-learning inference loss for Lag-llama and PatchTST. The central claim is that the Wasserstein distance is an informative indicator of foundation-model inference performance, with correlations above 0.60.

Significance. If the central claim were fully substantiated, the paper would offer a simple, model-agnostic, and computationally efficient tool for time-series dataset selection and model performance estimation. The application of the closed-form Wasserstein distance between Gaussian distributions to estimated parameters is mathematically standard, and the O(NL^2) complexity with L << N is attractive. The paper also provides a useful qualitative demonstration that the distance groups datasets from the same domain. However, the empirical support for the headline correlation claim is currently incomplete: the main text presents only two selected source datasets per model, the appendix does not summarize the full distribution of correlations, and the paper itself concedes that some source datasets yield negative correlations. Because the MVN/i.i.d. assumption is unvalidated and is used to explain failures, the robustness of the distance-loss relationship remains an open question.

major comments (3)
  1. [Sec. 4.3, Figs. 5-6, Appendix A.1, Sec. 4.4] The abstract and Section 4.3 claim 'high correlations between our proposed measure and the inference loss (>0.60)', but the main text supports this with only two selected source datasets per model (traffic-nips and electricity-nips). The complete results in Figs. 8-9 are scatter plots with no numerical summary, and Section 4.4 explicitly acknowledges that some source datasets, e.g., uber-tlc-hourly in Fig. 7, have negative correlations. As written, the headline number may reflect a favorable selection rather than general behavior. Please report the full distribution of correlation coefficients (e.g., Pearson and Spearman) across all 30 source datasets, for both models and both OOD and TL setups, and state whether the >0.60 claim refers to the median, the minimum, or only the selected subset.
  2. [Sec. 3.2, Sec. 4.4, Sec. 5.1] The method rests on the assumption that L-length windows are i.i.d. draws from an MVN, but the paper provides no validation of this assumption and Section 5.1 concedes that it may not hold for non-i.i.d. data. Section 4.4 attributes negative correlations to 'deviations of the dataset from the MVN assumption', yet the paper never measures those deviations or quantifies their impact. Since the Gaussian Wasserstein distance is still a well-defined moment-based statistic for non-Gaussian data, this assumption may not be fatal, but the paper needs either (a) a formal check of the MVN/i.i.d. assumption on the actual sampled windows, or (b) a sensitivity comparison with a nonparametric distance (e.g., empirical or sliced Wasserstein distance) to show that the distance-loss relationship is not an artifact of the Gaussian approximation. Without this, the explanation in Section 4.4 is unfalsifiable.
  3. [Sec. 4.3, Sec. 3.1] The reported correlations are obtained for a single choice of the free parameters L=48 and N=20,000, and the sampling process itself is not fully specified. Because the distance is computed from estimated means and covariances, both L and N influence the distance values and therefore the correlation with inference loss. Please report a sensitivity analysis over a range of L and N (or at least a second setting) to show that the claimed performance indicator is not sensitive to these user-chosen parameters. In addition, specify how the 20,000 windows are drawn (with or without replacement, whether overlapping windows are allowed, and the random seed) so that the experiment is reproducible.
minor comments (5)
  1. [Eq. (5)] The matrix square root in the covariance term is not defined; the formula is standard for two Gaussian distributions, but the notation should specify the principal square root, e.g., (Sigma_X^{1/2} Sigma_Y Sigma_X^{1/2})^{1/2}, to avoid ambiguity when Sigma_X and Sigma_Y do not commute.
  2. [Sec. 4.3] The paper states 'strong correlations (>0.60)' without specifying the correlation coefficient used. Please state whether Pearson or Spearman correlation is reported, and whether the values are averaged over models, setups, or source datasets.
  3. [Sec. 4.2] The claim that the Wasserstein heatmap is 'notably more informative' than the Euclidean and DTW heatmaps is based on visual inspection only. A quantitative criterion, such as agreement with domain labels or the correlation with inference loss for the baseline distances, would strengthen the comparison.
  4. [Sec. 4.4, Sec. 4.2.2] There are minor wording issues: 'covariate matrices' should be 'covariance matrices', and in Section 4.3 'our approach servers' should be 'our approach serves'. Additionally, the formatting of M^2 and the reference to 'Fig. 8- 9' should be cleaned up.
  5. [Appendix A.1] Figs. 8 and 9 contain the complete results but no numerical correlation coefficients; adding a table with the correlation for every source dataset, model, and setup would make the appendix directly usable and would resolve the main concern about selection bias.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Wasserstein distance is computed directly from the data, and the reported correlation with inference loss is an empirical observation rather than a quantity forced by construction.

full rationale

The derivation chain is self-contained: Sec. 3.2 estimates the MVN parameters from the sampled windows (Eq. 1), Sec. 3.3 defines the distance as the closed-form Wasserstein distance between the two estimated Gaussians (Eq. 5), and Sec. 4.3 compares that distance to measured inference losses. No parameter in the distance is fitted to the loss, so the >0.60 correlations cannot be a fitted input renamed as a prediction. The Mehra et al. [2024] citation in Sec. 1 is a background self-citation used to motivate the transfer-performance assumption, but the paper supplies its own direct empirical test of that assumption, so it is not load-bearing circularity. The manuscript itself flags the genuine limitations: Sec. 5.1 concedes that the i.i.d./MVN assumption is unvalidated for non-i.i.d. data, Sec. 4.4 reports negative correlations (e.g., uber-tlc-hourly, Fig. 7) and attributes them to MVN deviations, and the complete per-source correlation results (Figs. 8-9) are not summarized, so the 'representative' >0.60 figures are vulnerable to selection. These are issues of empirical support and reporting, not derivation-level circularity, because the distance itself is defined from data alone with no loss-derived fitting.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central distance computation rests on (1) the i.i.d. MVN assumption for windows, (2) hand-chosen sampling hyperparameters N and L, and (3) the hypothesis that the geometric distance is monotonically related to transfer performance. No new theoretical entities are introduced; the method reuses a standard distance on a new representation.

free parameters (3)
  • window length L = 48
    Chosen by hand; the distance values and the resulting correlations depend on this sampling window length.
  • sample size N = 20000
    Chosen by hand; larger N improves covariance estimates but increases compute. The same N is used for all datasets.
  • min-max normalization = per-dataset [0,1] scaling
    A hand-chosen preprocessing step that rescales each dataset and therefore directly affects the Wasserstein distances computed.
assumptions (3)
  • domain assumption Sampled time-series windows are i.i.d. draws from a multivariate normal distribution
    Central modeling assumption in Sec. 3.2; real time series exhibit autocorrelation and non-Gaussianity, acknowledged in Sec. 5.1.
  • domain assumption The 2-Wasserstein distance between the estimated MVNs is a meaningful measure of dataset similarity
    The paper's proposed definition; assumes first and second moments capture the structure relevant for transfer.
  • domain assumption Inference loss of foundation models is monotonically related to dataset distance
    The hypothesis driving the correlation experiments in Sec. 4.3; known to fail for some source datasets, e.g., uber-tlc-hourly (Fig. 7).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Time-Series Dataset Similarity using Wasserstein Distance." pith.science (2026). https://pith.science/paper/ANYNEV2L

@misc{pith2026250722189,
  author       = {Pith},
  title        = {Pith review of: Measuring Time-Series Dataset Similarity using Wasserstein Distance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ANYNEV2L}},
  note         = {Machine review of arXiv:2507.22189}
}
read the original abstract

The emergence of time-series foundation model research elevates the growing need to measure the (dis)similarity of time-series datasets. A time-series dataset similarity measure aids research in multiple ways, including model selection, finetuning, and visualization. In this paper, we propose a distribution-based method to measure time-series dataset similarity by leveraging the Wasserstein distance. We consider a time-series dataset an empirical instantiation of an underlying multivariate normal distribution (MVN). The similarity between two time-series datasets is thus computed as the Wasserstein distance between their corresponding MVNs. Comprehensive experiments and visualization show the effectiveness of our approach. Specifically, we show how the Wasserstein distance helps identify similar time-series datasets and facilitates inference performance estimation of foundation models in both out-of-distribution and transfer learning evaluation, with high correlations between our proposed measure and the inference loss (>0.60).

Figures

Figures reproduced from arXiv: 2507.22189 by the authors.

Figure 1
Figure 1. Foundation models are trained on source datasets and applied to target datasets. Presumably, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (Left) A heatmap of pairwise Wasserstein distances of thirty time-series datasets. The colorbar indicates the distance, where darker colors indicate higher similarities and closer distances. Wasserstein distance helps identify hierarchical clusters (highlighted in green) and subclusters (highlighted in red) of datasets. (Right) A visualization using the computed Wasserstein distances is rendered based on a force-dir… view at source ↗
Figure 3
Figure 3. A comparison of distance heatmaps of various approaches: (Left) Our Wasserstein [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Additional heatmaps of clustering-based distances for comparison: (Left) minimum [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Out-Of-Distribution (OOD) and Transfer Learning (TL) inference losses of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Out-Of-Distribution (OOD) and Transfer Learning (TL) inference losses of [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Out-Of-Distribution (OOD) inference loss of [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Out-Of-Distribution (OOD) and Transfer Learning (TL) inference losses of [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Out-Of-Distribution (OOD) and Transfer Learning (TL) inference losses of [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 canonical work pages

  1. [9]

    Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation.arXiv preprint arXiv:2406.19049,

    Amartya Sanyal, Yaxi Hu, Yaodong Yu, Yian Ma, Yixin Wang, and Bernhard Schölkopf. Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation.arXiv preprint arXiv:2406.19049,

    arXiv

  2. [10]

    Financial time series forecasting with deep learning: A systematic literature review: 2005–2019.Applied soft computing, 90:106181,

    Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. Financial time series forecasting with deep learning: A systematic literature review: 2005–2019.Applied soft computing, 90:106181,

    2005

  3. [11]

    Research on healthy anomaly detection model based on deep learning from multiple time-series physiological signals.Scientific Programming, 2016(1):5642856,

    Kai Wang, Youjin Zhao, Qingyu Xiong, Min Fan, Guotan Sun, Longkun Ma, and Tong Liu. Research on healthy anomaly detection model based on deep learning from multiple time-series physiological signals.Scientific Programming, 2016(1):5642856,

    work page 2016

  4. [13]

    URLhttps://doi.org/10.24963/ijcai.2024/186

    doi: 10.24963/ijcai.2024/186. URLhttps://doi.org/10.24963/ijcai.2024/186. Main Track. Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7856–7865,

    doi:10.24963/ijcai.2024/186 2024

  5. [1989]

    Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts.arXiv preprint arXiv:2202.06523,

    Weixin Liang and James Zou. Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts.arXiv preprint arXiv:2202.06523,

    arXiv

  6. [2012]

    Don’t stop pretraining: Adapt language models to domains and tasks.arXiv preprint arXiv:2004.10964,

    Suchin Gururangan, Ana Marasovi ´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks.arXiv preprint arXiv:2004.10964,

    arXiv 2004

  7. [2016]

    Similarity Between Two Stochastic Differential Systems

    Xiaoying Wang, Yuecai Han, and Yong Li. Similarity between two stochastic differential systems. arXiv preprint arXiv:2310.10901,

    work page Pith review arXiv

  8. [2017]

    A decoder-only foundation model for time-series forecasting.arXiv preprint arXiv:2310.10688,

    10 Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting.arXiv preprint arXiv:2310.10688,

    arXiv

Show all 13 references
  1. [2020]

    Chronos: Learning the language of time series.arXiv preprint arXiv:2403.07815,

    Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series.arXiv preprint arXiv:2403.07815,

    arXiv

  2. [2021]

    Gluonts: Probabilistic time series models in python.arXiv preprint arXiv:1906.05264,

    Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al. Gluonts: Probabilistic time series models in python.arXiv preprint arXiv:1906.05264,

    1906 arXiv

  3. [2023]

    Unitime: A language-empowered unified model for cross-domain time series forecasting

    Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowered unified model for cross-domain time series forecasting. In Proceedings of the ACM on Web Conference 2024, pages 4095–4106,

    2024

  4. [2024]

    A similarity measure of gaussian process predictive distributions

    Lucia Asencio Martín and Eduardo C Garrido-Merchán. A similarity measure of gaussian process predictive distributions. InAdvances in Artificial Intelligence: 19th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2020/2021, Málaga, Spain, September 22–2...

    2020

  5. [2025]

    Statistical optimal transport.arXiv preprint arXiv:2407.18163,

    Sinho Chewi, Jonathan Niles-Weed, and Philippe Rigollet. Statistical optimal transport.arXiv preprint arXiv:2407.18163,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.