{"id":"49ee8abc-8ba5-405b-9390-9fa115269dc5","arxiv_id":"2412.09346","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PROM and TSMD-Bench give time series motif discovery a quantitative, generally applicable scoring method and a realistic benchmark built from real data.","lead":"This paper introduces PROM, a scoring method that checks both how many true repeating patterns a time series algorithm finds and how many wrong ones it reports. It also builds a new practice benchmark from real sensor data, making it possible to compare motif-discovery algorithms fairly and at scale.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PROM's Lemma 1 depends on non-overlapping GT motif sets; with overlapping or nested GT segments a discovered segment can match two GT segments, making the matching order-dependent and contradicting the 'broadly applicable' claim.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the non-overlap constraint on GT motif sets is the premise for Lemma 1 and therefore for the entire matching logic of PROM. Within the stated setting, the mathematical machinery is sound: Lemma 1 is correct, the greedy individual matching is order-independent, and the Hungarian set-level matching is clearly specified. The concern is about scope, not internal consistency. The abstract and introduction repeatedly describe PROM as 'broadly applicable,' which is undermined if realistic ground truths allow overlapping or nested motif segments. The proposed counterexample demonstrates exactly how Lemma 1 breaks: a single discovered segment can have overlap rate greater than 0.5 with two overlapping GT segments, so the greedy matching result and the final F1 depend on processing order. TSMD-Bench is constructed to satisfy the non-overlap assumption, so the empirical evaluation cannot detect this failure. The reader's other concerns, such as the ARI-based dataset selection and the lack of external validation for 'more comprehensive,' are secondary and do not affect the metric's validity within its defined scope. I therefore agree with the reader's conditional verdict and propose a concrete test that would settle whether the broad-applicability claim needs qualification.","tokens_in":16562,"tokens_out":24376,"duration_ms":262931,"concrete_test":"Run PROM on the minimal counterexample: G={G1={[1:10]}, G2={[6:15]}}, M={M1={[1:15]}}. Compute F1 twice, processing GT segments in orders ([1:10],[6:15]) and ([6:15],[1:10]). If the two F1 values differ, PROM is order-dependent on overlapping ground truth, so the metric is not broadly applicable beyond the non-overlap assumption; the paper's claim should be qualified accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 restricts GT so that 'segments from different motif sets cannot overlap.' This is not a cosmetic restriction: Lemma 1 (Section 4.1) uses it to show every discovered segment is matchable with at most one GT segment, and the greedy per-GT matching is order-independent and total-OR maximizing. If the restriction is lifted, the lemma fails. Example: GT segments from two motif sets, beta1=[1:10] and beta2=[6:15], and a discovered segment alpha=[1:15]. Then OR(alpha,beta1)=OR(alpha,beta2)=10/15>0.5, so alpha is matchable with both. Processing GT segments in order beta1 then beta2 matches alpha to beta1 and leaves beta2 unmatched; reversed order gives the opposite. The contingency table and hence P, R, and F1 change. The paper is internally consistent because it explicitly assumes non-overlap, but the abstract's 'broadly applicable' claim overstates the setting: nested motifs (e.g., a recurring sub-shape inside a larger recurring pattern) or concurrent overlapping activities occur in real applications, and TSMD-Bench satisfies the assumption by construction, so the experiments never probe this boundary. This is a scope limitation of the central claim rather than a defect in the metric within its stated setting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PROM, a precision-recall-F1-style metric for evaluating time series motif discovery (TSMD) results against ground-truth motif sets, and TSMD-Bench, a set of synthetic benchmark time series built from UCR/UEA classification instances. PROM matches discovered motifs to ground-truth motifs by overlap rate, matches discovered motif sets to ground-truth sets via the Hungarian algorithm, and derives micro- or macro-averaged precision, recall, and F1 from the resulting matching matrix. The benchmark is constructed by concatenating class instances from easily clusterable classification datasets. Experiments with 11 TSMD methods on 14 benchmark datasets compare PROM with existing metrics, analyze method performance, and show that random-walk-based benchmarks are too easy.","tokens_in":16796,"tokens_out":12926,"duration_ms":121912,"significance":"If PROM is adopted, it would give the TSMD community a standard evaluation tool that handles multiple, variable-length, and multivariate motif sets while penalizing both motif-level and set-level false positives and negatives. The paper is careful: the matching procedure is formally defined with a proof of Lemma 1, hyperparameters are tuned on validation sets, and rankings are compared with Kendall's tau and statistical significance tests. The benchmark construction is a useful new resource. The main caveats are that the metric assumes non-overlapping ground-truth motif sets, which limits the 'broadly applicable' claim, and that a scalability claim is unsubstantiated.","major_comments":[{"comment":"The non-overlap assumption on ground-truth motif sets (Section 2: 'segments from different motif sets cannot overlap') is load-bearing: Lemma 1 and the order-independence of the individual matching procedure both rely on it. If ground-truth segments from different sets are allowed to overlap or nest, a discovered segment can be matchable with two ground-truth segments (e.g., β1=[1:10], β2=[6:15], α=[1:15] gives OR(α,β1)=OR(α,β2)=10/15>0.5), making the greedy matching order-dependent and the contingency table change. The paper is internally consistent, but the abstract and Section 1 claim that PROM is 'broadly applicable' and 'overcomes those limitations' of existing metrics. Since real-world TSMD applications can involve overlapping or nested recurring patterns, I recommend that the authors either extend the matching to handle overlapping ground-truth segments, or add an explicit limitation statement and restrict the 'broadly applicable' claim to non-overlapping ground-truth settings.","section":"Section 2, Section 4.1, Lemma 1"},{"comment":"The statement that PROM has time complexity O(km + m^3) in the illustrative case g=d=m with k motifs per set is not justified by the algorithm as described. The individual matching step in Section 4.1 requires, for each ground-truth segment, finding the discovered segment with highest overlap rate; without an additional data structure this is O((mk)^2) comparisons, yielding O(m^2 k^2 + m^3) overall. The authors should either describe an efficient implementation that achieves the claimed bound or correct the complexity claim, since the current text suggests PROM scales linearly in the time series length, which the described procedure does not guarantee.","section":"Section 6.1.1"}],"minor_comments":[{"comment":"The text refers to the 'UCR and UAE Time Series Classification Archive'; the archive is UCR/UEA. Please correct the typo (also in the same paragraph, 'UKAC' appears to be a misspelling).","section":"Section 5.2"},{"comment":"The proof of Lemma 1 can be made more explicit: if β1 and β2 were disjoint, their intersections with α would be disjoint subsets of α, so |α∩β1|+|α∩β2| ≤ |α|, contradicting each being greater than 0.5|α|.","section":"Section 4.1"},{"comment":"The text says 'The bottom right entry M*_{d+1,g+1} is undefined', but since M* is (g+1)×(d+1), the bottom-right entry should be M*_{g+1,d+1}. Please fix the index order.","section":"Section 4.2"},{"comment":"The critical difference diagrams would be more informative if they included the critical difference value and the exact post-hoc test used (e.g., Nemenyi), so that the statistical significance claims can be assessed.","section":"Figure 12"}],"recommendation":"major_revision","confidential_remarks":"The authors evaluate their own method (LoCoMotif) with their own metric and benchmark. The protocol is standard (validation-based tuning), and the result is not implausible, but I would encourage the editor to keep in mind the potential conflict when interpreting the comparison. The main scientific issue is the scope limitation of PROM; the complexity claim is a separate technical correction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PROM and TSMD-Bench are a solid, reproducible contribution to TSMD evaluation. The headline result is real: the field gets a formally defined precision/recall metric that handles variable-length, multivariate, multi-set cases, and a benchmark built from real UCR/UEA data instead of random-walk plus inserted instances.\n\nWhat's actually new: the overlap criterion itself appeared in their prior LoCoMotif paper, but this paper is the first to formalize it as a metric—Lemma 1, the matching matrix, and the complexity analysis are original. TSMD-Bench is a genuinely new benchmark construction, and the 11-method comparison is the largest systematic evaluation I've seen in this area. The random-walk triviality result (a Dickey-Fuller-based method gets F1=0.98 on random-walk benchmarks) convincingly backs their claim that those benchmarks are too easy.\n\nThe math holds up. Lemma 1 is correct given the stated assumption that ground-truth segments from different motif sets cannot overlap, and the assignment step is standard. The experimental setup is careful: validation-based hyperparameter tuning, statistical tests via critical difference diagrams, and 30,800 results. They also ship code, which is good practice.\n\nSoft spots, in proportion. The main one is the load-bearing assumption of non-overlapping GT motif sets. The abstract says 'broadly applicable,' but that's an overstatement. If GT motif sets can overlap or nest, a discovered segment can be matchable with two GT segments, and the greedy matching becomes order-dependent—exactly the example in the stress-test note. The authors state the assumption clearly in Section 2, so it's not an internal inconsistency, but the framing oversells the scope. The benchmark is constructed to satisfy the assumption, so the experiments never test how much this matters in practice. I'd like to see a discussion of when the assumption holds and possibly an extension or a principled restriction.\n\nSecond, the benchmark dataset selection uses ARI >= 0.5, which biases toward easily clusterable datasets. That doesn't invalidate the benchmark, but it's worth stating as a design choice with consequences. Third, the claim that PROM is 'more comprehensive' is supported by design arguments and examples, not an external validation. That's normal for a metric paper, so I wouldn't hold it against them. Fourth, the authors evaluate their own method with their own metric—disclosed, but the rankings should be read with that in mind.\n\nOverall, this deserves a serious referee. The contributions are useful and the flaws are addressable. I'd recommend accepting it with revisions that clarify the scope and possibly add a discussion of overlapping ground truth.","headline":"PROM and TSMD-Bench are a solid, reproducible contribution to TSMD evaluation, but the 'broadly applicable' claim is narrower than advertised because of the non-overlap assumption.","tokens_in":17371,"tokens_out":2599,"would_cite":true,"duration_ms":23829,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PROM is a metric that evaluates time series motif discovery by matching discovered motifs to ground-truth motifs by overlap, then optimally matching the motif sets, and finally computing precision, recall, and F1 from the resulting…","keywords":["time series motif discovery","quantitative evaluation","precision recall","optimal matching","benchmark construction","variable-length motifs","multivariate time series"],"falsifier":"Take a time series whose ground truth contains two motif sets that overlap in time, run PROM on a fixed discovered set, and see whether the reported precision and recall change when the ground-truth segments are processed in the reverse order; because Lemma 1 fails in that case, any order-dependence would falsify the claim that PROM is a broadly applicable, order-independent metric.","tokens_in":16338,"feed_emoji":"📊","tokens_out":9677,"duration_ms":87132,"temperature":0.7,"pith_summary":"This paper claims that quantitative evaluation of time series motif discovery has been limited by metrics that assume fixed-length motifs, a single motif set, or motif pairs, and often ignore false discoveries. It introduces PROM, which compares any discovered collection of motif sets against a ground truth of motif sets in three steps: match individual motifs when their time overlap exceeds half, optimally match motif sets with an assignment algorithm, and count true and false positives and negatives to produce precision, recall, and F1. The paper also builds TSMD-Bench, fourteen benchmark datasets assembled from real time series classification data, which it claims are more representative and harder than the random-walk benchmarks used previously. Experiments with eleven methods are presented as evidence that older metrics skew toward either recall or precision, while PROM measures both, and that large-scale systematic comparison of motif discovery methods is now possible.","feed_headline":"New metric scores time-series motifs on precision and recall","feed_subtitle":"PROM matches discovered motifs to ground truth, then counts false positives and negatives, making methods comparable.","key_machinery":"The carrying mechanism is a two-level matching pipeline with a named identity at its core. For two segments the overlap rate is $\\mathrm{OR}(\\alpha, \\beta) = \\frac{|\\alpha \\cap \\beta|}{|\\alpha \\cup \\beta|}$, and segments are matchable when $\\mathrm{OR} > 0.5$. Lemma 1 states that because ground-truth segments never overlap, each discovered segment is matchable with at most one ground-truth segment; this makes the per-motif matching order-independent and maximizes total overlap. The per-motif match counts form a contingency table, whose columns are then permuted by the Hungarian method to maximize the diagonal sum, giving an optimal matching of motif sets. Precision, recall, and F1 are micro-averaged over the matching matrix, with an option to ignore false-positive motif sets so that well-conserved but off-target patterns are not penalized.","core_discovery":"The central claim is that a single criterion, PROM, can quantitatively evaluate any time series motif discovery output in the general setting of multiple motif sets, arbitrary cardinality, and variable-length motifs. PROM's definition reduces evaluation to a two-level matching: per-motif matching by an overlap rate threshold, then a per-set assignment solved as a linear sum assignment problem. From the resulting matching matrix it derives true positives, false negatives, and false positives at both the motif and the set level, and these yield micro-averaged precision, recall, and F1. The paper shows that on 30,800 results generated by eleven methods over fourteen datasets, PROM ranks results differently from existing correctness and score metrics, which the authors attribute to those metrics measuring mostly recall or mostly precision.","pith_inferences":["Beyond the paper: the two-level matching at the heart of PROM is generic enough to apply to other structured prediction tasks where the output is a set of sets with instance-level matching, such as event detection or pattern mining, since it only relies on overlap and optimal assignment.","Beyond the paper: the choice to ignore false-positive motif sets is a tuning decision that can change rankings; a fair comparison protocol would need a fixed default or a two-score report with and without penalizing off-target sets.","Beyond the paper: TSMD-Bench's construction from real instances may still create unlabeled higher-order patterns, since two classes can share a sub-shape; quantifying how often such off-target patterns appear would clarify how clean the ground truth actually is.","Beyond the paper: if overlapping or nested ground-truth motifs become important in an application, PROM would need to replace its non-overlap assumption with a one-to-many matching scheme, trading simplicity for broader applicability."],"forward_implications":["PROM produces a precision, recall, and F1 score for any time series motif discovery result, so methods that differ in the number of motif sets, motif length, or dimensionality can be compared on the same scale.","The benchmark construction converts any easily clusterable time-series classification dataset into a time series motif discovery benchmark with labeled ground truth, yielding fourteen datasets with separate validation and test splits.","Because the correctness metric mainly tracks recall and the score metric mainly tracks precision, a method comparison that relies on either one alone can be misleading; reporting PROM's precision and recall together avoids the bias.","A simple detector that finds random-walk regions achieves nearly perfect F1 on random-walk-based synthetic benchmarks, which supports the claim that those benchmarks are too easy to be representative.","With PROM and TSMD-Bench, model selection and hyperparameter tuning for time series motif discovery methods can be done automatically through F1 on validation sets, enabling large-scale systematic performance comparisons."],"supporting_citations":[{"why":"Supplies the overlap-based correctness metric that PROM is compared against and that does not penalize false-positive motifs.","marker":"Yingchareonthawornchai et al. (2013)"},{"why":"Supplies the score metric and the fixed-set precision/recall definitions that PROM extends to multiple and variable-length motif sets.","marker":"Bagnall et al. (2014)"},{"why":"Provides the assignment algorithm used to match discovered motif sets to ground-truth sets optimally.","marker":"Jonker and Volgenant (1987)"},{"why":"Is the UCR/UEA archive from which the classification datasets for TSMD-Bench are drawn.","marker":"Dau et al. (2018)"},{"why":"Provides the clustering benchmark results used to select easily clusterable univariate classification datasets.","marker":"Javed et al. (2020)"},{"why":"Introduces LoCoMotif, the method that first used the PROM criterion and that is evaluated with it here.","marker":"Van Wesenbeeck et al. (2024)"},{"why":"Motivates the option to ignore off-target discovered motif sets with the calibration-signal example.","marker":"Dau and Keogh (2017)"},{"why":"Supplies the statistical ranking tests used to compare methods on PROM scores.","marker":"Benavoli et al. (2016)"},{"why":"Provides the Dickey-Fuller test used by the random-walk detector to show random-walk benchmarks are easy.","marker":"Harris (1992)"}],"fun_headline_variants":["New metric PROM benchmarks time-series motif discovery","PROM: a fairer ruler for time-series motif methods","Time-series motif discovery gets a quantitative benchmark","PROM and TSMD-Bench measure motif quality precisely","New benchmark and metric rank motif-discovery methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The metric assumes the ground-truth motif sets are made of non-overlapping segments, so no segment of the time series belongs to two true motifs; if real ground truths contain overlapping or nested motifs, the matching guarantee behind PROM no longer holds.","fun_headline_variants_meta":{"raw":{"variants":["New metric PROM benchmarks time-series motif discovery","PROM: a fairer ruler for time-series motif methods","Time-series motif discovery gets a quantitative benchmark","PROM and TSMD-Bench measure motif quality precisely","New benchmark and metric rank motif-discovery methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1159,"prompt_tokens":832,"completion_tokens":327,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":254}},"tokens_in":448,"tokens_out":327,"duration_ms":3790,"temperature":1.0,"reasoning_tokens":254,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:05:39.827283+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a time series whose ground truth contains two motif sets that overlap in time, run PROM on a fixed discovered set, and see whether the reported precision and recall change when the ground-truth segments are processed in the reverse order; because Lemma 1 fails in that case, any order-dependence would falsify the claim that PROM is a broadly applicable, order-independent metric.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the overlap-based correctness metric that PROM is compared against and that does not penalize false-positive motifs."},{"cited_title":"M., Zhu, Y., Gharghabi, S., Ratanamahatana, C","cited_arxiv_id":null,"evidence_quote":"Is the UCR/UEA archive from which the classification datasets for TSMD-Bench are drawn."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the statistical ranking tests used to compare methods on PROM scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Dickey-Fuller test used by the random-walk detector to show random-walk benchmarks are easy."}],"review_version":1}