{"id":"adf4fd22-2660-490a-86a1-2dd3207c059f","arxiv_id":"2505.21160","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A benchmark with 10 datasets and 13 transformations ranks 41 time-series evaluation measures by reliability, consistency, and speed, and finds embedding choice strongly affects scores.","lead":"STEB is a benchmark that ranks 41 existing measures for comparing synthetic time series to real data, using 10 datasets and 13 controlled transformations. It shows that the choice of embedding model changes scores dramatically, so standardized embeddings are needed for trustworthy evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reliability ranking is direction-dependent: rrel assumes every measure is higher-is-better, so lower-is-better measures such as WD, KLD, and predictive score are systematically penalized regardless of their actual quality.","rationale":"The reader's weakest assumption identified the a priori expected-behavior table and the monotonicity criterion as load-bearing. My stress-test found a more specific, concrete defect inside that step: the rrel formulas do not respect each measure's own score orientation. The paper states at the outset that higher is better, but it never normalizes or filters the 41 measures to satisfy that assumption. This is not merely a disagreement with consensus; it is an internal inconsistency between the definitions of the measures (many are distances or errors) and the reliability formula. The proposed test is decisive: recomputing rrel with corrected comparison directions for lower-is-better measures would either confirm that the rankings change, in which case the central ranking must be revised, or show that the ranking is robust, in which case the concern is resolved. I therefore keep the reader's CONDITIONAL verdict, but the condition should explicitly require the orientation analysis before the ranking claims are accepted.","tokens_in":33961,"tokens_out":8237,"duration_ms":100365,"concrete_test":"Recompute rrel for every lower-is-better measure (at minimum WD, KLD, JSD, distributional metric, predictive score, TRTS, TSTR, temporal correlation, spatial correlation, and CT) using the opposite comparison direction: for expected 'Worsen' count si < sj, and for expected 'Improve' count si > sj, then rebuild Tables 1 and 6. If the top-ranked measures or the relative ordering change materially, the published ranking is an artifact of the higher-is-better assumption. As a minimal sanity check, verify that WD increases monotonically with kappa under Gaussian noise on a scaled dataset; if STEB's current 'Worsen' rule would mark that behavior as a violation, the orientation bug is confirmed directly.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"STEB's central output is the reliability ranking in Tables 1 and 6, built on rrel as defined in Section D.1. For the expected behavior 'Worsen', Eq. 12's counterpart counts score pairs with si > sj, i.e., it treats a decreasing raw score as the signature of worsening quality. For 'Improve' it counts si < sj. This is only valid if every measure is oriented so that larger scores mean better quality. The paper explicitly relies on this in Section 3.1: 'we expect s to get worse with increasing kappa' and 'assumption: higher is better'. No score-direction normalization is described anywhere in Sections 3.4 or 3.5. Yet many of the 41 measures are naturally lower-is-better: WD, KLD, JSD, distributional metric, predictive score, TRTS, TSTR, temporal/spatial correlation, and CT are distances/errors/divergences. For such a measure under Gaussian noise, a fidelity-aware response should be an increase in raw score as kappa grows; the 'Worsen' rule instead counts a decrease as correct, so the measure looks unreliable even when it is behaving exactly as its definition implies. The paper's own MAE/miMAE example exposes the problem: regular MAE 'would satisfy none of these inequalities, resulting in bad performance', while its monotone inverse miMAE is rewarded. Since a monotone rescaling preserves all information about the measure, rrel is not actually measuring reliability; it is measuring agreement with an arbitrary higher-is-better convention. The consistency indicator inherits this bias because it is computed on rrel. Consequently, the central claim that the benchmark can rank evaluation measures by reliability is not supported unless the orientation of each measure is accounted for.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces STEB, a benchmark framework for evaluating quantitative measures of synthetic time series quality. STEB applies 13 data transformations along a modulation path indexed by intensity κ, computes scores of 41 measures over 10 datasets and multiple random seeds, and aggregates them into a reliability indicator rrel (the fraction of score pairs that move in a direction expected a priori), a consistency indicator rcon (based on pairwise Kolmogorov–Smirnov tests across seeds/datasets), and running-time statistics. The main empirical claims are a ranking of measures in four quality categories (fidelity, generalization, privacy, representativeness) and a side result that the choice of TS embedding (TS2Vec, Catch22, Concat) strongly affects scores. The paper also documents engineering contributions: parallel execution, caching, failure recovery, and open-source plans.","tokens_in":34274,"tokens_out":4743,"duration_ms":58015,"significance":"The paper addresses a real need: there is no large-scale, measure-centric benchmark for synthetic time series evaluation, and the comparison of 41 measures on 10 datasets is a substantial empirical effort. The engineering infrastructure (parallel workers, caching, failure tracking, time accounting) and the transparency about failed tests are valuable. The embedding experiment is a useful cautionary result that is largely independent of the reliability-ranking methodology. If the reliability ranking were based on an orientation-invariant and validated criterion, STEB would be a useful community resource. As it stands, the ranking is not yet a trustworthy basis for measure selection.","major_comments":[{"comment":"The reliability indicator rrel is computed under the explicit assumption that higher scores are better (Section 3.1: 'assumption: higher is better'), but no score-direction normalization is applied for measures that are naturally lower-is-better. For the 'Worsen' case, Eq. (12) counts pairs with si > sj; a lower-is-better measure such as WD, KLD, JSD, distributional metric, predictive score, TRTS, TSTR, or correlation-based distances will correctly produce increasing raw scores as quality worsens, and rrel will count those pairs as wrong. The paper's own MAE/miMAE example in Section 3.1 demonstrates the problem: regular MAE 'would satisfy none of these inequalities, resulting in bad performance', although MAE and miMAE are related by a fixed monotone transformation and therefore carry identical information. Consequently, Tables 1 and 6 rank measures partly by their orientation convention rather than by their reliability. The authors should either normalize score directions (e.g., by estimating each measure's orientation on a known-good vs known-bad setting, or by defining an absolute/symmetric indicator), or report two rankings and re-evaluate which conclusions survive.","section":"Section 3.1, Section 3.5, Eq. (12)"},{"comment":"The expected behaviors in Table 3 are asserted to be 'confidently assigned a priori', but they are the sole ground truth against which rrel is computed. This makes rrel a measure of conformity to the authors' expectations, not an independent measure of reliability. Several entries are debatable (e.g., Gaussian noise is assumed to improve generalization and privacy while worsening fidelity and representativeness; mode dropping is assumed to leave fidelity unchanged), and if any expectation is wrong, all downstream rankings inherit the error. The manuscript should validate the table, for example by (i) soliciting independent annotations from multiple experts and reporting inter-annotator agreement, (ii) providing a formal justification or a reference for each entry, and (iii) performing a sensitivity analysis in which disputed entries are flipped to show how rankings change. Without this, the central ranking is circular in the sense that 'good' is defined as 'matches Table 3'.","section":"Section 3.5, Table 3"},{"comment":"The consistency indicator rcon is computed as the fraction of pairwise Kolmogorov–Smirnov tests that fail to reject the null hypothesis of equal rrel distributions across groups of 10 seeds (or 10 datasets). With only 10 samples per group, the KS test has low power, which likely explains why the seed consistency column in Table 7 is 1.0 for almost every measure and category; such a result is uninformative. The authors should report an effect-size-based or calibrated measure of dispersion (e.g., variance of rrel across seeds/datasets, or a power analysis showing what differences the KS test can detect at n=10), and should avoid interpreting rcon=1.0 as evidence of high consistency.","section":"Section D.2, Eq. (23), Table 7"}],"minor_comments":[{"comment":"There are several typos: 'Number os statistically different bins' should be 'Number of statistically different bins', 'incomming nearest neighbor distance' should be 'incoming nearest neighbor distance', and Table 1 (right) uses 'DOMAIS' while the rest of the paper uses 'DOMIAS'.","section":"Section A"},{"comment":"The 'Constant, real' formula in Eq. (17) divides by the median µ; if µ = 0, the indicator is undefined or unstable. Since many scores are bounded below by zero, this case should be handled explicitly.","section":"Section D.1, Eq. (17)"},{"comment":"The MAPE/PCC panel is difficult to interpret because MAPE is capped at 10 and 51 of 61 cases exceed 1000%. A supplementary figure showing the score distributions (e.g., box plots per embedder) would make the magnitude of the embedding effect much clearer.","section":"Section 5.2, Table 1 (right)"},{"comment":"The running-time table separates cache-aided from un-aided executions only in the long tables in the appendix; the main Table 8 mixes both, and several measures (e.g., WCS, MTop-Div) have high variance or missing entries. It would help to state explicitly in the main text how caching affects the reported averages.","section":"Section 5.1, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The orientation problem is severe and affects the central ranking; the paper needs a re-analysis with orientation-invariant indicators or explicit per-measure orientation normalization. I would not reject the paper outright because the framework and the embedding experiment have clear value, but the reliability ranking should not be published in its current form. The authors should also consider releasing code and raw results before acceptance to support reproducibility and independent checking."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead STEB (arXiv:2505.21160). The scope is right and the work is substantial: 41 measures, 10 datasets, 13 transformations, running-time tables, and a side experiment showing the embedding model massively changes scores. That side experiment alone is worth having; the benchmark framework itself, with modular transformations and caching, will likely be useful.\n\nThe main reliability ranking is not yet defensible. The stress-test note is correct. rrel for \"Worsen\" counts score pairs with si > sj, and for \"Improve\" counts si < sj, under an explicit \"higher is better\" assumption. Many measures in the list are naturally lower-is-better: WD, KLD, JSD, distributional metric, predictive score, TRTS/TSTR, correlation differences, CT. For those measures a fidelity-worsening transformation produces an increasing raw score, which STEB counts as failure to detect the change. The paper's own MAE/miMAE example is the tell: regular MAE \"satisfies none of these inequalities\" while its monotone inverse is rewarded. A monotone rescaling preserves all information, so rrel is measuring agreement with a direction convention, not reliability. No score-direction normalization appears in Sections 3.4 or 3.5. This systematically biases the rankings and the consistency indicator, which is computed on rrel.\n\nThe hand-assigned expected behaviors in Table 3 are a second concern. They may be reasonable, but \"can be confidently assigned a priori\" is doing a lot of work; the ranking is by construction a measure of conformity to that table. I would not call it fatal on its own — every benchmark needs a reference point — but combined with the direction issue it means the headline ranking should not be used as-is.\n\nSmaller issues: the seed-consistency KS test almost always returns 1.0, which may reflect low power rather than genuine robustness; no code or data is released in the preprint, and the paper says it will be open-sourced after acceptance, which blocks reproduction.\n\nThe transformations and category taxonomy are sensible, the running-time data is honest, and the embedding experiment is a clean empirical result. The paper is a serious contribution in need of a specific fix. I would send it to review, with a strong request to orient each measure (or use relative/percent change) and to re-run the rankings, and to make code/data available.\n\nFor a reading group, it is a good case study in benchmark methodology. I would cite it for the embedding result, not for the ranking.","headline":"Useful first broad benchmark, but the reliability ranking is built on an unhandled higher-is-better convention that systematically penalizes distance/error measures; fixable, but central as is.","tokens_in":34852,"tokens_out":2851,"would_cite":true,"duration_ms":34748,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a benchmark that ranks 41 synthetic-time-series quality measures by reliability, consistency, and speed, and finds the choice of embedding model changes scores so strongly that cross-embedding comparisons are not…","keywords":["synthetic time series","evaluation benchmark","quality measures","reliability indicator","consistency indicator","distribution modulation","embedding choice","time series transformations"],"falsifier":"Ask a panel of time series experts to order the datasets produced by one transformation, say Gaussian noise at $\\kappa=0.0,0.1,\\ldots,1.0$, from best to worst in fidelity and privacy; if the panel's ordering disagrees with Table 3 on any pair of steps, then the a priori expectations are not universally shared and the $r_{\\mathrm{rel}}$ indicator does not measure what it claims.","tokens_in":33739,"feed_emoji":"📊","tokens_out":8643,"duration_ms":88565,"temperature":0.7,"pith_summary":"The paper is trying to solve a selection problem: with dozens of quantitative measures for synthetic time series quality, nobody knows which measures actually track quality. Its proposal, STEB, replaces a generative model with a set of controllable data transformations and an intensity slider, so that every test has a known expected direction of score change. Across 41 measures, ten datasets, and four quality aspects (fidelity, generalization, privacy, representativeness), it converts each measure's agreement with those expectations into a reliability score, adds a consistency score and running-time measurements, and publishes the resulting ranking. The strongest finding is that the upstream time series embedding has a decisive effect on the final score, which means the embedding must be fixed before any comparison of synthetic data can be trusted.","feed_headline":"Embedding choice can swing synthetic time series scores","feed_subtitle":"A benchmark of 41 quality measures shows evaluators must fix one embedding or scores are not comparable.","key_machinery":"The central mechanism is controlled distribution modulation: instead of using a trained generator, STEB applies a transformation $T$ to the real dataset with an intensity parameter $\\kappa$, producing a modulation path $D_\\kappa$ for $\\kappa = 0, 0.1, \\ldots, 1.0$ whose expected effect on each quality category is known in advance. The reliability indicator $r_{\\mathrm{rel}}$ is the fraction of score pairs $(s_i, s_j)$, $i<j$, for which a measure moves in the expected direction—improve, worsen, or stay constant—and this fraction, averaged over datasets, transformations, and random seeds, is the paper's primary yardstick. The consistency indicator $r_{\\mathrm{con}}$ adds a check that the measure's reliability does not depend on the random seed or dataset.","core_discovery":"On its own terms, the paper establishes that evaluation measures for synthetic time series can be compared objectively by testing them on pseudo-synthetic data with known properties. STEB applies 13 transformations (noise, label corruption, misalignment, mode collapse, wavelet changes, and others) at eleven intensity levels, and for each measure counts the fraction of score pairs that move in the direction Table 3 declares a priori for that quality category. The resulting indicator $r_{\\mathrm{rel}}$ ranks α-Precision first for fidelity and representativeness, ACS first for generalization, and autocorrelation first for privacy, while showing no measure covers all categories. A separate experiment on 24 embedder-dependent measures quantifies the embedding effect: most pairwise score changes between a deep representation, a classical feature-based embedder, and simple channel concatenation exceed 1000%, so the paper concludes that the choice of time series embedding has significant impact on any synthetic-data evaluation.","pith_inferences":["If the embedding effect observed for 24 measures extends to the remaining 17, published results that used different embeddings may not be comparable even when the same measure was used; re-running the benchmark under a fixed embedding would be the direct test.","The reliability criterion equates informativeness with monotonicity along the modulation path, so a measure that is informative but non-monotone (for instance, one that saturates or responds only at high distortion) would be scored as unreliable even though it may carry useful signal.","The a priori expectations in Table 3 are treated as ground truth, but they could themselves be validated by an independent human study; if expectations shift, the ranking of measures shifts with them.","The same modulation-path template could be carried to other data modalities, such as tabular or image synthesis, by swapping in a new transformation set and a new expected-behavior table."],"forward_implications":["Any comparison of synthetic time series generators should fix the embedding model first; otherwise the measured difference between generators may be an artifact of the embedding.","No single measure covers all four quality aspects, so a practical evaluation suite needs at least two measures, with the paper suggesting α-Precision for fidelity, ACS for generalization, autocorrelation for privacy, and Context-FID for representativeness when compute is not the binding constraint.","Measures that perform well on fidelity and representativeness tend to perform poorly on generalization and privacy, which means category-specific selection matters.","The running-time tables give users a concrete way to trade reliability against compute, and measures with high failure rates (such as DOMIAS in the main experiment) can be avoided even before testing.","Because STEB is extensible, the ranking can be recomputed as new measures, transformations, and datasets are added, so the benchmark is intended to track the state of the art rather than settle it once."],"supporting_citations":[{"why":"Supplies the modulation-path idea: distorting data by a controllable factor to test evaluation measures.","marker":"[34]"},{"why":"The closest experimental precedent, manipulating data to study measure behavior; its note on embedding dependence motivates the side experiment.","marker":"[19]"},{"why":"Defines several measures used in the benchmark, such as the discriminative and predictive scores.","marker":"[60]"},{"why":"The survey that catalogs the 41 measures and their definitions, supplying the measure pool for the benchmark.","marker":"[52]"},{"why":"An existing benchmark framework that takes evaluation measures as given, which STEB is designed to inform.","marker":"[45]"},{"why":"An existing time series generation framework whose integrated measures are accepted without assessment, a gap STEB fills.","marker":"[41]"},{"why":"The deep embedding model used as default; the embedding experiment shows changing it strongly changes scores.","marker":"[61]"},{"why":"The non-deep feature-based embedder used in the comparison of embedding effects.","marker":"[33]"}],"fun_headline_variants":["New benchmark tests 41 synthetic time series metrics","Synthetic data scores hinge on embedding choice","Benchmark reveals embedding impact on synthetic quality","First benchmark for synthetic time series evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole ranking rests on the table of a priori expected behaviors—for instance, that Gaussian noise improves privacy and generalization while worsening fidelity—so if any of those expectations is wrong or debatable, the reliability indicator will misrank the very measures it is designed to judge.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark tests 41 synthetic time series metrics","Synthetic data scores hinge on embedding choice","Benchmark reveals embedding impact on synthetic quality","First benchmark for synthetic time series evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1396,"prompt_tokens":849,"completion_tokens":547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":492}},"tokens_in":465,"tokens_out":547,"duration_ms":6184,"temperature":1.0,"reasoning_tokens":492,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:33:15.126564+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a panel of time series experts to order the datasets produced by one transformation, say Gaussian noise at $\\kappa=0.0,0.1,\\ldots,1.0$, from best to worst in fidelity and privacy; if the panel's ordering disagrees with Table 3 on any pair of steps, then the a priori expectations are not universally shared and the $r_{\\mathrm{rel}}$ indicator does not measure what it claims.","supporting_citations":[{"cited_title":"Are gans created equal? a large-scale study","cited_arxiv_id":null,"evidence_quote":"Supplies the modulation-path idea: distorting data by a controllable factor to test evaluation measures."},{"cited_title":"An empirical study on evaluation metrics of generative adversarial networks.arXiv preprint, August 2018","cited_arxiv_id":null,"evidence_quote":"The closest experimental precedent, manipulating data to study measure behavior; its note on embedding dependence motivates the side experiment."},{"cited_title":"Evaluation is key: a survey on evaluation measures for synthetic time series.Journal of Big Data, 11(1):66, 2024","cited_arxiv_id":null,"evidence_quote":"The survey that catalogs the 41 measures and their definitions, supplying the measure pool for the benchmark."},{"cited_title":"Synthcity: a benchmark framework for diverse use cases of tabular synthetic data","cited_arxiv_id":null,"evidence_quote":"An existing benchmark framework that takes evaluation measures as given, which STEB is designed to inform."},{"cited_title":"Tsgm: A flexible framework for generative modeling of synthetic time series","cited_arxiv_id":null,"evidence_quote":"An existing time series generation framework whose integrated measures are accepted without assessment, a gap STEB fills."},{"cited_title":"Ts2vec: Towards universal representation of time series.Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):8980–8987, June 2022","cited_arxiv_id":null,"evidence_quote":"The deep embedding model used as default; the embedding experiment shows changing it strongly changes scores."},{"cited_title":"catch22: CAnonical Time-series CHaracteristics: Selected through highly comparative time-series analysis.Data Mining and Knowledge Discovery, 33(6):1821–1852, 2019","cited_arxiv_id":null,"evidence_quote":"The non-deep feature-based embedder used in the comparison of embedding effects."}],"review_version":1}