Pith. sign in

REVIEW 3 major objections 4 minor 70 references

STEB: In Search of the Best Evaluation Approach for Synthetic Time Series

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces a benchmark that ranks 41 synthetic-time-series quality measures by reliability, consistency, and speed, and finds the choice of embedding model changes scores so strongly that cross-embedding comparisons are not…

desk verdict Useful first broad benchmark, but the reliability ranking is built on an unhandled higher-is-better convention that systematically penalizes distance/error measures; fixable, but central as is. read the letter →

arxiv 2505.21160 v1 pith:Z2N4P22R submitted 2025-05-27 cs.LG cs.AI

Michael Stenger , Robert Leppich , André Bauer , Samuel Kounev This is my paper · ORCID
classification cs.LGcs.AI
keywords synthetictimeseriesevaluationbenchmarkqualitymeasuresreliabilityindicatorconsistencydistributionmodulationembeddingchoicetransformations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to solve a selection problem: with dozens of quantitative measures for synthetic time series quality, nobody knows which measures actually track quality. Its proposal, STEB, replaces a generative model with a set of controllable data transformations and an intensity slider, so that every test has a known expected direction of score change. Across 41 measures, ten datasets, and four quality aspects (fidelity, generalization, privacy, representativeness), it converts each measure's agreement with those expectations into a reliability score, adds a consistency score and running-time measurements, and publishes the resulting ranking. The strongest finding is that the upstream time series embedding has a decisive effect on the final score, which means the embedding must be fixed before any comparison of synthetic data can be trusted.

What carries the argument

The central mechanism is controlled distribution modulation: instead of using a trained generator, STEB applies a transformation $T$ to the real dataset with an intensity parameter $\kappa$, producing a modulation path $D_\kappa$ for $\kappa = 0, 0.1, \ldots, 1.0$ whose expected effect on each quality category is known in advance. The reliability indicator $r_{\mathrm{rel}}$ is the fraction of score pairs $(s_i, s_j)$, $i<j$, for which a measure moves in the expected direction—improve, worsen, or stay constant—and this fraction, averaged over datasets, transformations, and random seeds, is the paper's primary yardstick. The consistency indicator $r_{\mathrm{con}}$ adds a check that the measure's reliability does not depend on the random seed or dataset.

What would settle it

Ask a panel of time series experts to order the datasets produced by one transformation, say Gaussian noise at $\kappa=0.0,0.1,\ldots,1.0$, from best to worst in fidelity and privacy; if the panel's ordering disagrees with Table 3 on any pair of steps, then the a priori expectations are not universally shared and the $r_{\mathrm{rel}}$ indicator does not measure what it claims.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that evaluation measures for synthetic time series can be compared objectively by testing them on pseudo-synthetic data with known properties. STEB applies 13 transformations (noise, label corruption, misalignment, mode collapse, wavelet changes, and others) at eleven intensity levels, and for each measure counts the fraction of score pairs that move in the direction Table 3 declares a priori for that quality category. The resulting indicator $r_{\mathrm{rel}}$ ranks α-Precision first for fidelity and representativeness, ACS first for generalization, and autocorrelation first for privacy, while showing no measure covers all categories. A separate experiment on 24 embedder-dependent measures quantifies the embedding effect: most pairwise score changes between a deep representation, a classical feature-based embedder, and simple channel concatenation exceed 1000%, so the paper concludes that the choice of time series embedding has significant impact on any synthetic-data evaluation.

Load-bearing premise

The whole ranking rests on the table of a priori expected behaviors—for instance, that Gaussian noise improves privacy and generalization while worsening fidelity—so if any of those expectations is wrong or debatable, the reliability indicator will misrank the very measures it is designed to judge.

Editorial extensions

If this is right

  • Any comparison of synthetic time series generators should fix the embedding model first; otherwise the measured difference between generators may be an artifact of the embedding.
  • No single measure covers all four quality aspects, so a practical evaluation suite needs at least two measures, with the paper suggesting α-Precision for fidelity, ACS for generalization, autocorrelation for privacy, and Context-FID for representativeness when compute is not the binding constraint.
  • Measures that perform well on fidelity and representativeness tend to perform poorly on generalization and privacy, which means category-specific selection matters.
  • The running-time tables give users a concrete way to trade reliability against compute, and measures with high failure rates (such as DOMIAS in the main experiment) can be avoided even before testing.
  • Because STEB is extensible, the ranking can be recomputed as new measures, transformations, and datasets are added, so the benchmark is intended to track the state of the art rather than settle it once.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the embedding effect observed for 24 measures extends to the remaining 17, published results that used different embeddings may not be comparable even when the same measure was used; re-running the benchmark under a fixed embedding would be the direct test.
  • The reliability criterion equates informativeness with monotonicity along the modulation path, so a measure that is informative but non-monotone (for instance, one that saturates or responds only at high distortion) would be scored as unreliable even though it may carry useful signal.
  • The a priori expectations in Table 3 are treated as ground truth, but they could themselves be validated by an independent human study; if expectations shift, the ranking of measures shifts with them.
  • The same modulation-path template could be carried to other data modalities, such as tabular or image synthesis, by swapping in a new transformation set and a new expected-behavior table.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces STEB, a benchmark framework for evaluating quantitative measures of synthetic time series quality. STEB applies 13 data transformations along a modulation path indexed by intensity κ, computes scores of 41 measures over 10 datasets and multiple random seeds, and aggregates them into a reliability indicator rrel (the fraction of score pairs that move in a direction expected a priori), a consistency indicator rcon (based on pairwise Kolmogorov–Smirnov tests across seeds/datasets), and running-time statistics. The main empirical claims are a ranking of measures in four quality categories (fidelity, generalization, privacy, representativeness) and a side result that the choice of TS embedding (TS2Vec, Catch22, Concat) strongly affects scores. The paper also documents engineering contributions: parallel execution, caching, failure recovery, and open-source plans.

Significance. The paper addresses a real need: there is no large-scale, measure-centric benchmark for synthetic time series evaluation, and the comparison of 41 measures on 10 datasets is a substantial empirical effort. The engineering infrastructure (parallel workers, caching, failure tracking, time accounting) and the transparency about failed tests are valuable. The embedding experiment is a useful cautionary result that is largely independent of the reliability-ranking methodology. If the reliability ranking were based on an orientation-invariant and validated criterion, STEB would be a useful community resource. As it stands, the ranking is not yet a trustworthy basis for measure selection.

major comments (3)
  1. [Section 3.1, Section 3.5, Eq. (12)] The reliability indicator rrel is computed under the explicit assumption that higher scores are better (Section 3.1: 'assumption: higher is better'), but no score-direction normalization is applied for measures that are naturally lower-is-better. For the 'Worsen' case, Eq. (12) counts pairs with si > sj; a lower-is-better measure such as WD, KLD, JSD, distributional metric, predictive score, TRTS, TSTR, or correlation-based distances will correctly produce increasing raw scores as quality worsens, and rrel will count those pairs as wrong. The paper's own MAE/miMAE example in Section 3.1 demonstrates the problem: regular MAE 'would satisfy none of these inequalities, resulting in bad performance', although MAE and miMAE are related by a fixed monotone transformation and therefore carry identical information. Consequently, Tables 1 and 6 rank measures partly by their orientation convention rather than by their reliability. The authors should either normalize score directions (e.g., by estimating each measure's orientation on a known-good vs known-bad setting, or by defining an absolute/symmetric indicator), or report two rankings and re-evaluate which conclusions survive.
  2. [Section 3.5, Table 3] The expected behaviors in Table 3 are asserted to be 'confidently assigned a priori', but they are the sole ground truth against which rrel is computed. This makes rrel a measure of conformity to the authors' expectations, not an independent measure of reliability. Several entries are debatable (e.g., Gaussian noise is assumed to improve generalization and privacy while worsening fidelity and representativeness; mode dropping is assumed to leave fidelity unchanged), and if any expectation is wrong, all downstream rankings inherit the error. The manuscript should validate the table, for example by (i) soliciting independent annotations from multiple experts and reporting inter-annotator agreement, (ii) providing a formal justification or a reference for each entry, and (iii) performing a sensitivity analysis in which disputed entries are flipped to show how rankings change. Without this, the central ranking is circular in the sense that 'good' is defined as 'matches Table 3'.
  3. [Section D.2, Eq. (23), Table 7] The consistency indicator rcon is computed as the fraction of pairwise Kolmogorov–Smirnov tests that fail to reject the null hypothesis of equal rrel distributions across groups of 10 seeds (or 10 datasets). With only 10 samples per group, the KS test has low power, which likely explains why the seed consistency column in Table 7 is 1.0 for almost every measure and category; such a result is uninformative. The authors should report an effect-size-based or calibrated measure of dispersion (e.g., variance of rrel across seeds/datasets, or a power analysis showing what differences the KS test can detect at n=10), and should avoid interpreting rcon=1.0 as evidence of high consistency.
minor comments (4)
  1. [Section A] There are several typos: 'Number os statistically different bins' should be 'Number of statistically different bins', 'incomming nearest neighbor distance' should be 'incoming nearest neighbor distance', and Table 1 (right) uses 'DOMAIS' while the rest of the paper uses 'DOMIAS'.
  2. [Section D.1, Eq. (17)] The 'Constant, real' formula in Eq. (17) divides by the median µ; if µ = 0, the indicator is undefined or unstable. Since many scores are bounded below by zero, this case should be handled explicitly.
  3. [Section 5.2, Table 1 (right)] The MAPE/PCC panel is difficult to interpret because MAPE is capped at 10 and 51 of 61 cases exceed 1000%. A supplementary figure showing the score distributions (e.g., box plots per embedder) would make the magnitude of the embedding effect much clearer.
  4. [Section 5.1, Table 8] The running-time table separates cache-aided from un-aided executions only in the long tables in the appendix; the main Table 8 mixes both, and several measures (e.g., WCS, MTop-Div) have high variance or missing entries. It would help to state explicitly in the main text how caching affects the reported averages.

Circularity Check

1 steps flagged · score 5.0 of 10

The reliability ranking reduces by construction to agreement with the paper's own a priori expectations in Table 3, and the 'higher is better' convention makes lower-is-better measures score as unreliable by definition.

  1. self definitional [Section 3.5, Eq. (12) and Table 3]
    "These expected behaviors can be confidently assigned a priori. The result is listed in Table 3. If the behavior is defined, we can compute a task-specific indicator rrel(t) ... Assuming si ∈ R and improvement means to increase si, we define rrel as the fraction of score pairs (si, sj), i < j with si < sj."

    rrel is defined as the fraction of score pairs that match the authors' a priori directions in Table 3. The paper's central output, the reliability ranking of 41 measures (Tables 1 and 6), is therefore by construction a ranking of agreement with those stated expectations. A measure is called 'reliable' exactly when its scores move in the direction the authors assumed, so the headline result restates the input expectations rather than testing them independently. The MAE/miMAE example in Section 3.1 makes the reduction explicit: miMAE is the inverse of MAE, so the same information yields opposite rrel values; the 'bad performance' of MAE is a consequence of the higher-is-better definition, not of any independent property of the measure.

full rationale

The central reliability ranking is defined by Eq. (12) and its Worsen counterpart as the fraction of score pairs satisfying the authors' a priori expected directions from Table 3. Thus the ranking of measures is, by the paper's own equations, a conformity score to those expectations. This is a real but partial circularity: the paper does not derive reliability from an external ground truth; it equates reliability with 'matches Table 3'. The paper's own example, where regular MAE scores badly while its inverse miMAE scores well, shows that the indicator is not invariant to monotone rescoring, so the result is driven by the arbitrary higher-is-better convention and by the a priori directions rather than by any intrinsic property of the measures. The direction issue also means lower-is-better measures such as WD, KLD, and predictive score are systematically penalized under the Worsen rule, which counts si > sj as the signature of worsening quality. This is a validity threat to the ranking, though it is a consequence of the same definitional choice. The paper is not otherwise circular: the embedding side experiment (Section 5.2) is an independent empirical comparison of TS2Vec, Catch22, and Concat, and there is no load-bearing self-citation chain that forces the conclusions. Because the main reliability ranking is definitionally tied to the paper's own expectations, but the embedding finding and running-time measurements retain independent content, a moderate circularity score of 5 is appropriate.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark framework STEB is a new software artifact, but it is not an invented physical entity of the kind this ledger tracks. The load-bearing assumptions are the a priori expected-behavior table and the monotonicity criterion, both domain assumptions, plus the four-category quality decomposition and the statistical validity of the consistency test. There are no fitted parameters in the usual sense, but several hand-chosen design values (epsilon, k, moving-average factor, segment length) influence the results.

free parameters (4)
  • epsilon (epsilon) in Constant reliability = 0.05
    Section D.1, Eq. 17: scores are considered constant if |mu - s_i| / mu <= epsilon. The 5% threshold is chosen by hand and directly controls rrel for constant-expected transformations.
  • Number of modulation steps k = 11 (kappa = 0.0, 0.1, ..., 1.0)
    Section 4: 'we use eleven equally spaced steps'. The number of score pairs in Eq. 12 grows as k^2, affecting the resolution of the reliability indicator.
  • Moving average filter width factor a = 1/3 if l >= 30, else 1
    Section 3.2: 'a*l*kappa + 1 wide where a = 1/3 if l >= 30 else a = 1'. This hand-set factor changes the smoothing intensity and thus what the transformation tests.
  • Segment length range in segment leaking = between l/4 and l/2
    Section 3.2: 'A segment is one channel of a TS subsequence and between l/4 and l/2 long'. Uniform random choice in this range is a design parameter.
assumptions (4)
  • domain assumption The 13 transformations induce the category-specific expected behaviors listed in Table 3 (e.g., Gaussian noise worsens fidelity and representativeness while improving generalization and privacy).
    Section 3.5 states these behaviors 'can be confidently assigned a priori'. The reliability indicator rrel is computed as the fraction of score pairs that match these expectations, so the entire ranking depends on this table being correct.
  • domain assumption A good evaluation measure should produce scores that monotonically degrade as transformation intensity kappa increases.
    Section 3.1 assumes 'we expect s to get worse with increasing kappa' and defines reliability as the fraction of monotone pairs (Eq. 12). Non-monotone but informative measures are penalized.
  • domain assumption The four categories (fidelity, generalization, privacy, representativeness) are a sufficient decomposition of synthetic data quality.
    Section 3.4, following [55, 1, 57], breaks quality into these four aspects. The whole evaluation is category-specific, and any quality aspect outside these four is not measured.
  • standard math The KS two-sample test, used to define consistency, correctly identifies equal distributions with the small group sizes available.
    Section D.2 applies KS to pairs of rrel groups (one per seed or dataset). With only a small number of rrel values per group, the test has low power, so 'consistent' results may be an artifact of low sensitivity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of STEB: In Search of the Best Evaluation Approach for Synthetic Time Series." pith.science (2026). https://pith.science/paper/Z2N4P22R

@misc{pith2026250521160,
  author       = {Pith},
  title        = {Pith review of: STEB: In Search of the Best Evaluation Approach for Synthetic Time Series},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2N4P22R}},
  note         = {Machine review of arXiv:2505.21160}
}
read the original abstract

The growing need for synthetic time series, due to data augmentation or privacy regulations, has led to numerous generative models, frameworks, and evaluation measures alike. Objectively comparing these measures on a large scale remains an open challenge. We propose the Synthetic Time series Evaluation Benchmark (STEB) -- the first benchmark framework that enables comprehensive and interpretable automated comparisons of synthetic time series evaluation measures. Using 10 diverse datasets, randomness injection, and 13 configurable data transformations, STEB computes indicators for measure reliability and score consistency. It tracks running time, test errors, and features sequential and parallel modes of operation. In our experiments, we determine a ranking of 41 measures from literature and confirm that the choice of upstream time series embedding heavily impacts the final score.

Figures

Figures reproduced from arXiv: 2505.21160 by the authors.

Figure 1
Figure 1. Depiction of the modulation concept. By modulating parameter [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Architectural design of STEB. Input (top [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Critical difference diagram for reliability indicator [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Critical difference diagram for reliability indicator [PITH_FULL_IMAGE:figures/full_fig_p026_4.png]
Figure 5
Figure 5. Figure 5: Critical difference diagram for reliability indicator [PITH_FULL_IMAGE:figures/full_fig_p027_5.png]
Figure 6
Figure 6. Figure 6: Critical difference diagram for reliability indicator [PITH_FULL_IMAGE:figures/full_fig_p028_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 67 canonical work pages

  1. [1]

    Saveliev, and Mihaela van der Schaar

    Ahmed Alaa, Boris Van Breugel, Evgeny S. Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? Sample-level metrics for evaluating and auditing generative models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors,Proceedings of the 39th International Conference on Machine Learning,...

    work page 2022

  2. [2]

    Yihao Ang, Qiang Huang, Yifan Bao, Anthony K. H. Tung, and Zhiyong Huang. Tsgbench: Time series generation benchmark.Proc. VLDB Endow., 17(3):305–318, November 2023

    work page 2023

  3. [3]

    Visual evaluation of generative adversarial networks for time series data.arXiv preprint, December 2019

    Hiba Arnout, Johannes Kehrer, Johanna Bronner, and Thomas Runkler. Visual evaluation of generative adversarial networks for time series data.arXiv preprint, December 2019

    work page 2019

  4. [4]

    Permutation entropy: a natural complexity measure for time series

    Christoph Bandt and Bernd Pompe. Permutation entropy: a natural complexity measure for time series. Physical review letters, 88(17):174102, 2002

    work page 2002

  5. [5]

    Manifold topology divergence: a framework for comparing data manifolds

    Serguei Barannikov, Ilya Trofimov, Grigorii Sotnikov, Ekaterina Trimbach, Alexander Korotin, Alexander Filippov, and Evgeny Burnaev. Manifold topology divergence: a framework for comparing data manifolds. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, pa...

    work page 2021

  6. [6]

    Using dynamic time warping to find patterns in time series

    Donald J Berndt and James Clifford. Using dynamic time warping to find patterns in time series. In AAAI-94 Workshop on Knowledge Discovery in Databases, volume 10, pages 359–370, Menlo Park, California, 1994. The AAAI Press

    work page 1994

  7. [7]

    Bousseljot, D

    R. Bousseljot, D. Kreiseler, and A. Schnabel. Nutzung der EKG-Signaldatenbank CARDIODAT der PTB über das Internet, January 1995

    work page 1995

  8. [8]

    Generative adversarial networks in time series: A systematic literature review.ACM Comput

    Eoin Brophy, Zhengwei Wang, Qi She, and Tomás Ward. Generative adversarial networks in time series: A systematic literature review.ACM Comput. Surv., 55(10), February 2023

    work page 2023

Show all 70 references
  1. [9]

    Candanedo, Véronique Feldheim, and Dominique Deramaix

    Luis M. Candanedo, Véronique Feldheim, and Dominique Deramaix. Data driven prediction models of energy use of appliances in a low-energy house.Energy and Buildings, 140:81–97, 2017

    2017

  2. [10]

    Stl: A seasonal-trend decomposition.J

    Robert B Cleveland, William S Cleveland, Jean E McRae, and Irma Terpenning. Stl: A seasonal-trend decomposition.J. off. Stat, 6(1):3–73, 1990

    1990

  3. [11]

    The ucr time series classification archive, October 2018

    Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdul- lah Mueen, Gustavo Batista, and Hexagon-ML. The ucr time series classification archive, Octob...

    2018

  4. [12]

    Statistical comparisons of classifiers over multiple data sets.Journal of Machine Learning Research, 7:1–30, 2006

    Janez Demšar. Statistical comparisons of classifiers over multiple data sets.Journal of Machine Learning Research, 7:1–30, 2006

    2006

  5. [13]

    Hyland, and Gunnar Rätsch

    Cristóbal Esteban, Stephanie L. Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans.arXiv preprint, June 2017

    2017

  6. [14]

    Tabular and latent space synthetic data generation: a literature review

    Joao Fonseca and Fernando Bacao. Tabular and latent space synthetic data generation: a literature review. Journal of Big Data, 10(1):115, 2023

    2023

  7. [15]

    Unsupervised scalable representation learning for multivariate time series

    Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Syste...

    2019

  8. [16]

    A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. Ch. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals.Circulation [Onlin...

    2000

  9. [17]

    A survey on deep learning for human activity recognition.ACM Comput

    Fuqiang Gu, Mu-Huan Chung, Mark Chignell, Shahrokh Valaee, Baoding Zhou, and Xue Liu. A survey on deep learning for human activity recognition.ACM Comput. Surv., 54(8), October 2021. 10

    2021

  10. [18]

    The significance probability of the smirnov two-sample test.Arkiv för matematik, 3(5):469– 486, 1958

    J L Hodges Jr. The significance probability of the smirnov two-sample test.Arkiv för matematik, 3(5):469– 486, 1958

    1958

  11. [19]

    An empirical study on evaluation metrics of generative adversarial networks.arXiv preprint, August 2018

    Gao Huang, Yang Yuan, Qiantong Xu, Chuan Guo, Yu Sun, Felix Wu, and Kilian Weinberger. An empirical study on evaluation metrics of generative adversarial networks.arXiv preprint, August 2018

    2018

  12. [20]

    Establishing a unified evaluation framework for human motion generation: A comparative analysis of metrics.arXiv preprint, 2024

    Ali Ismail-Fawaz, Maxime Devanne, Stefano Berretti, Jonathan Weber, and Germain Forestier. Establishing a unified evaluation framework for human motion generation: A comparative analysis of metrics.arXiv preprint, 2024

    2024

  13. [21]

    PSA-GAN: Progressive self attention GANs for synthetic time series

    Paul Jeha, Michael Bohlke-Schneider, Pedro Mercado, Shubham Kapoor, Rajbir Singh Nirwan, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. PSA-GAN: Progressive self attention GANs for synthetic time series. InInternational Conference on Learning Representations, 2022

    2022

  14. [22]

    Ai in healthcare: time-series forecasting using statistical, neural, and ensemble architectures.Frontiers in big data, 3:4, 2020

    Shruti Kaushik, Abhinav Choudhury, Pankaj Kumar Sheron, Nataraj Dasgupta, Sayee Natarajan, Larry A Pickett, and Varun Dutt. Ai in healthcare: time-series forecasting using statistical, neural, and ensemble architectures.Frontiers in big data, 3:4, 2020

    2020

  15. [23]

    Nonlinearity, Information and Robotics

    Patrik Joslin Kenfack, Daniil Dmitrievich Arapov, Rasheed Hussain, S.M. Ahsan Kazmi, and Adil Khan. On the fairness of generative adversarial networks (gans). In2021 International Conference "Nonlinearity, Information and Robotics" (NIR), pages 1–7, 2021

    2021

  16. [24]

    Kruskal and W

    William H. Kruskal and W. Allen Wallis. Use of ranks in one-criterion variance analysis.Journal of the American Statistical Association, 47(260):583–621, 1952

    1952

  17. [25]

    Improved precision and recall metric for assessing generative models

    Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information P...

    2019

  18. [26]

    Modeling long- and short-term temporal patterns with deep neural networks

    Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. InThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, page 95–104. Association for Compu...

    2018

  19. [27]

    Lee, Ralf Gommers, Filip Waselewski, Kai Wohlfahrt, and Aaron O’Leary

    Gregory R. Lee, Ralf Gommers, Filip Waselewski, Kai Wohlfahrt, and Aaron O’Leary. Pywavelets: A python package for wavelet analysis.Journal of Open Source Software, 4(36):1237, 2019

    2019

  20. [28]

    [sok] the great gan bake off, an extensive systematic evaluation of generative adversarial network architectures for time series synthesis.Journal of Systems Research, 2(1), 2022

    Mark Leznik, Arne Lochner, Stefan Wesner, and Jörg Domaschka. [sok] the great gan bake off, an extensive systematic evaluation of generative adversarial network architectures for time series synthesis.Journal of Systems Research, 2(1), 2022

    2022

  21. [29]

    Tts-gan: A transformer-based time-series generative adversarial network

    Xiaomin Li, Vangelis Metsis, Huangyingrui Wang, and Anne Hee Hiong Ngu. Tts-gan: A transformer-based time-series generative adversarial network. In Martin Michalowski, Syed Sibte Raza Abidi, and Samina Abidi, editors,Artificial Intelligence in Medicine, pages 133–143, Cham, 20...

    2022

  22. [30]

    Tts-cgan: A transformer time-series conditional gan for biosignal data augmentation.arXiv preprint, June 2022

    Xiaomin Li, Anne Hee Hiong Ngu, and Vangelis Metsis. Tts-cgan: A transformer time-series conditional gan for biosignal data augmentation.arXiv preprint, June 2022

    2022

  23. [31]

    Revisiting classifier two-sample tests

    David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. InInternational Conference on Learning Representations, Toulon, France, April 2017

    2017

  24. [32]

    Pcf-gan: generating sequential data via the characteristic function of measures on the path space

    Hang Lou, Siran Li, and Hao Ni. Pcf-gan: generating sequential data via the characteristic function of measures on the path space. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, page...

    2023

  25. [33]

    catch22: CAnonical Time-series CHaracteristics: Selected through highly comparative time-series analysis.Data Mining and Knowledge Discovery, 33(6):1821–1852, 2019

    Carl H Lubba, Sarab S Sethi, Philip Knaute, Simon R Schultz, Ben D Fulcher, and Nick S Jones. catch22: CAnonical Time-series CHaracteristics: Selected through highly comparative time-series analysis.Data Mining and Knowledge Discovery, 33(6):1821–1852, 2019

    2019

  26. [34]

    Are gans created equal? a large-scale study

    Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors,Advances in Neural Information Processing Systems, volum...

    2018

  27. [35]

    Web traffic time series forecasting.Kaggle, 2017

    Maggie, Oren Anava, Vitaly Kuznetsov, and Will Cukierski. Web traffic time series forecasting.Kaggle, 2017. 11

    2017

  28. [36]

    H. B. Mann and D. R. Whitney. On a test of whether one of two random variables is stochastically larger than the other.The Annals of Mathematical Statistics, 18(1):50–60, 1947

    1947

  29. [37]

    A non-parametric test to detect data-copying in generative models

    Casey Meehan, Kamalika Chaudhuri, and Sanjoy Dasgupta. A non-parametric test to detect data-copying in generative models. In Silvia Chiappa and Roberto Calandra, editors,Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume ...

    2020

  30. [38]

    Unimib shar: A dataset for human activity recognition using acceleration data from smartphones.Applied Sciences, 7(10), 2017

    Daniela Micucci, Marco Mobilio, and Paolo Napoletano. Unimib shar: A dataset for human activity recognition using acceleration data from smartphones.Applied Sciences, 7(10), 2017

    2017

  31. [39]

    Reliable fidelity and diversity metrics for generative models

    Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In Hal Daumé III and Aarti Singh, editors,Proceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedi...

    2020

  32. [40]

    Conditional sig-wasserstein gans for time series generation.arXiv preprint, June 2020

    Hao Ni, Lukasz Szpruch, Magnus Wiese, Shujian Liao, and Baoren Xiao. Conditional sig-wasserstein gans for time series generation.arXiv preprint, June 2020

    2020

  33. [41]

    Tsgm: A flexible framework for generative modeling of synthetic time series

    Alexander Nikitin, Letizia Iannucci, and Samuel Kaski. Tsgm: A flexible framework for generative modeling of synthetic time series. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, ...

    2024

  34. [42]

    Gebremedhin

    Skyler Norgaard, Ramyar Saeedi, Keyvan Sasani, and Assefaw H. Gebremedhin. Synthetic sensor data generation for health applications: A supervised deep learning approach. In2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC),...

    2018

  35. [43]

    Rosenblum, and Wenzhuo Yang

    Kun Ouyang, Reza Shokri, David S. Rosenblum, and Wenzhuo Yang. A non-parametric generative model for human trajectories. InProceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI’18, page 3812–3817. AAAI Press, 2018

    2018

  36. [44]

    Marco A. F. Pimentel, Alistair E. W. Johnson, Peter H. Charlton, Drew Birrenkott, Peter J. Watkinson, Lionel Tarassenko, and David A. Clifton. Toward a robust estimation of respiratory rate from pulse oximeters.IEEE Transactions on Biomedical Engineering, 64(8):1914–1923, 2017

    1914

  37. [45]

    Synthcity: a benchmark framework for diverse use cases of tabular synthetic data

    Zhaozhi Qian, Rob Davis, and Mihaela van der Schaar. Synthcity: a benchmark framework for diverse use cases of tabular synthetic data. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, ...

    2023

  38. [46]

    Classification accuracy score for conditional generative models

    Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    2019

  39. [47]

    On gans and gmms

    Eitan Richardson and Yair Weiss. On gans and gmms. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018

    2018

  40. [48]

    Physiological time-series analysis using approximate entropy and sample entropy.American journal of physiology-heart and circulatory physiology, 278(6):H2039– H2049, 2000

    Joshua S Richman and J Randall Moorman. Physiological time-series analysis using approximate entropy and sample entropy.American journal of physiology-heart and circulatory physiology, 278(6):H2039– H2049, 2000

    2000

  41. [49]

    Generating multivariate time series with common source coordinated gan (cosci-gan).Advances in Neural Information Processing Systems, 35:32777–32788, 2022

    Ali Seyfi, Jean-Francois Rajotte, and Raymond Ng. Generating multivariate time series with common source coordinated gan (cosci-gan).Advances in Neural Information Processing Systems, 35:32777–32788, 2022

    2022

  42. [50]

    On characterizing gan convergence through proximal duality gap

    Sahil Sidheekh, Aroof Aimen, and Narayanan C Krishnan. On characterizing gan convergence through proximal duality gap. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Res...

    2021

  43. [51]

    Thinking in categories: A survey on assessing the quality for time series synthesis.J

    Michael Stenger, André Bauer, Thomas Prantl, Robert Leppich, Nathaniel Hudson, Kyle Chard, Ian Foster, and Samuel Kounev. Thinking in categories: A survey on assessing the quality for time series synthesis.J. Data and Information Quality, 16(2), June 2024

    2024

  44. [52]

    Evaluation is key: a survey on evaluation measures for synthetic time series.Journal of Big Data, 11(1):66, 2024

    Michael Stenger, Robert Leppich, Ian Foster, Samuel Kounev, and André Bauer. Evaluation is key: a survey on evaluation measures for synthetic time series.Journal of Big Data, 11(1):66, 2024. 12

    2024

  45. [53]

    A note on the evaluation of generative models

    Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. In International Conference on Learning Representations, 2016

    2016

  46. [54]

    Antropy, 2024

    Raphael Vallat. Antropy, 2024. Version 0.1.8

    2024

  47. [55]

    Decaf: Generating fair synthetic data using causally-aware generative networks

    Boris van Breugel, Trent Kyono, Jeroen Berrevoets, and Mihaela van der Schaar. Decaf: Generating fair synthetic data using causally-aware generative networks. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information...

    2021

  48. [56]

    Membership inference attacks against synthetic data through overfitting detection

    Boris van Breugel, Hao Sun, Zhaozhi Qian, and Mihaela van der Schaar. Membership inference attacks against synthetic data through overfitting detection. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors,Proceedings of The 26th International Conference on Art...

    2023

  49. [57]

    Beyond privacy: Navigating the opportunities and challenges of synthetic data.arXiv preprint, April 2023

    Boris van Breugel and Mihaela van der Schaar. Beyond privacy: Navigating the opportunities and challenges of synthetic data.arXiv preprint, April 2023

    2023

  50. [58]

    Deep hedging: Learning to simulate equity option markets.arXiv preprint, November 2019

    Magnus Wiese, Lianjun Bai, Ben Wood, and Hans Buehler. Deep hedging: Learning to simulate equity option markets.arXiv preprint, November 2019

    2019

  51. [59]

    Cot-gan: Generating sequential data via causal optimal transport

    Tianlin Xu, Li Kevin Wenliang, Michael Munn, and Beatrice Acciaio. Cot-gan: Generating sequential data via causal optimal transport. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages ...

    2020

  52. [60]

    Time-series generative adversarial networks

    Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative adversarial networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 32. Curran Associates,...

    2019

  53. [61]

    Ts2vec: Towards universal representation of time series.Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):8980–8987, June 2022

    Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series.Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):8980–8987, June 2022. A Summary of Evaluation Measure...

    2022

  54. [62]

    We chose a sample size ofn= 100

    subsample both datasets to speed up the computation due to the quadratic complexity. We chose a sample size ofn= 100. Authenticity*[ 1]. Authenticity seeks to measure the proportion of synthetic instances that are novel relative to the real dataset by comparing the distance of...

  55. [63]

    We chose a sample size ofn= 100, considerably more representative than the originaln= 10

    subsample Ds to speed up the computation due to the quadratic complexity. We chose a sample size ofn= 100, considerably more representative than the originaln= 10. Improved precision*[ 25]. This measure counts and averages the number of generated TS for which there is a real T...

  56. [64]

    To evaluate, we chose a sample size ofn= 100for both datasets

    subsample DR and use only one synthetic TS to monitor training. To evaluate, we chose a sample size ofn= 100for both datasets. JSD*[ 43]. Generally speaking, the Jensen-Shannon divergence (JSD) measures the dissimilarity between two distributions. In this case, these are the d...

  57. [65]

    Train on real-test on synthetic

    subsample DR and Ds to speed up the computation due to the quadratic complexity. We chose a sample size ofn= 100. Predictive score[ 60]. This measure trains a simple forecasting model on the generated time series to conduct one-step-ahead predictions. Then, it evaluates its pe...

    1990

  58. [66]

    Usually, one measure is needed per category

    Determine which categories are relevant for the given use case. Usually, one measure is needed per category. For each category, repeat all following steps

  59. [67]

    Start with the measure ranked highest reliability (see Table 6)

  60. [68]

    We suggest a minimum ofr con = 0.5

    If the measure lacks consistency in this category (see Table 7), move on to the next best measure. We suggest a minimum ofr con = 0.5

  61. [69]

    For this purpose, the running time on the dataset most similar in size, TS length, and number of feature channels to the use case’s target dataset should be considered

    If the use case dictates running time constraints, for instance, because many parametrizations of a generative model must be evaluated during optimization, measures with excessive running time should be skipped (see Table 8 in combination with Table 11). For this purpose, the ...

  62. [70]

    Similarly, some measures are prone to errors such as DOMIAS and CT , which requires additional effort and knowledge to fix (see Table 4 and Table 5)

    Further, in case ease-of-use is an important selection criterion, it makes sense to skip embedder-dependent measures or only use them in case embeddings can be reused in other categories. Similarly, some measures are prone to errors such as DOMIAS and CT , which requires addit...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.