{"id":"dac406b1-8f17-4bcc-a938-b083e13e945a","arxiv_id":"2412.06368","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Contrastive accuracy, the fraction of augmented sequence pairs that are mutual nearest neighbors in embedding space, correlates with downstream accuracy of time series foundation models in UCR experiments.","lead":"This paper introduces a label-free metric, contrastive accuracy, for judging whether a time series foundation model's pre-training data supports good downstream classification. The metric measures how often two augmented views of the same sequence become nearest neighbors in the model's embedding space, and the authors report that it correlates with accuracy across UCR datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Improvement-prediction evidence for the selection claim is statistically fragile: under the paper's own setup, correlations are mixed and often non-significant.","rationale":"I agree with the reader's overall CONDITIONAL verdict, but my load-bearing concern is more specific and more immediately checkable than the augmentation-transfer assumption. The reader's weakest assumption is that uniformity/alignment with RandomCropResize transfers across distribution shifts. That is a real external-validity worry. However, the improvement-prediction experiment already shows mixed and sometimes negative correlations under the same augmentation and the same UCR distribution used throughout the paper. This is an internal-consistency problem: the evidence for the selection use case, which is the paper's advertised practical contribution, is not robust even in the setting where the method should work. The Section 4.1 correlations are strong but do not test dataset selection; they mostly track subsample size. Therefore the decisive check is a pooled, properly clustered statistical analysis of the Section 4.2 data. If that analysis fails, no amount of augmentation tuning is likely to rescue the central claim; if it succeeds, the conditional acceptance can proceed. I would keep the verdict CONDITIONAL (UNCHANGED) because the proposed test could resolve the concern either way, and the current evidence is insufficient for acceptance or rejection.","tokens_in":9176,"tokens_out":14873,"duration_ms":161724,"concrete_test":"Pool all unique (X0, Xnew) pairs from Section 4.2, compute Spearman's rho between delta-Acon and delta-P, and construct a 95% confidence interval using a cluster bootstrap that resamples whole pre-training datasets. Report the fraction of the 12 fixed-X0 panels with positive rho, and repeat the correlation test using all per-run values instead of 5-run averages. If the pooled CI excludes zero and the effect is positive in most panels, the concern is resolved; otherwise the central selection claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central practical claim, that contrastive accuracy can serve as a label-free criterion to select or expand pre-training datasets, rests on the improvement-prediction experiment in Section 4.2. This is the only experiment that varies dataset identity rather than subsample size, so it is the only direct test of the 'quality' interpretation. Under the paper's own fixed augmentation and within the same UCR archive, the evidence is internally inconsistent. In the fix-X0 panels of Figure 5, only 6 of 12 datasets show correlations with p<0.05; EOGVerticalSignal is negative (rho=-0.07, p=0.83), and five others are non-significant at the 0.05 level (CricketY, MelbournePedestrian, Phoneme, ScreenType, WordSynonyms). The main text illustrates the claim with AllGestureWiimoteX, one of the strongest panels, rather than the full distribution of outcomes. The p-values are computed on 11 averaged points, each from 5 runs, so run-level variance and dataset clustering are ignored. The Section 4.1 subsampling experiment, while showing high correlations, confounds contrastive accuracy and downstream accuracy with the subsample ratio r, so it cannot establish that the metric ranks datasets by quality. Consequently, the abstract's claim that the measure 'can serve as a criterion to search for time series datasets' is not established by the reported evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether the quality of pre-training data for a time-series foundation model can be evaluated without labels. It proposes contrastive accuracy (Acon), the expected fraction of samples whose two augmented views are mutual nearest neighbours in the embedding space of a contrastively pre-trained encoder, and claims that Acon correlates with downstream accuracy. The paper reports two experiments on UCR datasets: one varying the subsample ratio of a single pre-training dataset (Section 4.1) and one measuring the correlation between Acon improvement and downstream accuracy improvement when adding a candidate dataset to the pre-training set (Section 4.2). On this basis it suggests that Acon can serve as a criterion to search for pre-training datasets.","tokens_in":9447,"tokens_out":5579,"duration_ms":53105,"significance":"If the correlation held robustly across datasets and augmentation families, Acon would be a cheap, label-free tool for selecting and expanding pre-training corpora for time-series foundation models, which is a practically important problem. The paper is clearly written, the metric is simple, and the evaluation uses a large benchmark (UCR) with multiple seeds. The central idea is plausible and the Section 4.1 results are encouraging. However, the evidence for the dataset-selection claim is currently mixed, and the main experiment that varies dataset identity shows several non-significant and one negative correlation; this needs to be addressed before the central claim is accepted. The metric is model-dependent and augmentation-dependent, which is acknowledged in the paper but limits the current scope.","major_comments":[{"comment":"The improvement-prediction experiment, which is the only direct test of the dataset-selection claim, does not consistently support it. In the 'Fix X0, Vary Xnew' panels of Figure 5, only 6 of 12 datasets reach p<0.05; EOGVerticalSignal has rho=-0.07 (p=0.83), and CricketY, MelbournePedestrian, Phoneme, ScreenType, and WordSynonyms are not significant at the 0.05 level. The text illustrates the claim with AllGestureWiimoteX (rho=0.74, p=0.0088), which is one of the strongest panels, without reporting the full distribution. Moreover, the reported p-values appear to be computed on 11 per-dataset averaged points (each the mean of 5 runs), so run-level variance and dataset clustering are ignored; the analysis should be repeated on the individual runs or with a mixed-effects model, and the full set of correlations should be reported.","section":"Sec. 4.2 / Figs. 5-6"},{"comment":"The strong correlations between Acon and downstream accuracy are measured while varying the subsampling ratio r of the same pre-training dataset. In this design Acon, Ptrain, and Ptest all increase with r, so the correlation may be driven by the common dependence on dataset size rather than by data quality. This experiment therefore does not demonstrate that Acon ranks different datasets by quality, which is the claimed use case. A control experiment with fixed-size subsets of different datasets, or a partial correlation controlling for r, is needed to support the interpretation.","section":"Sec. 4.1 / Fig. 1 and Fig. 4"},{"comment":"The definition of Acon is under-specified. The similarity vector si in Eq. (2) is defined for a batch B used in training, while Eq. (3) applies the argmax over all n examples in X' and states that for large n the data are split into disjoint batches. It is not specified how the indicator and the expectation over augmentations are aggregated over these batches, nor how the mutual-neighbour criterion is computed for examples at batch boundaries. This makes the metric difficult to reproduce exactly. In addition, Acon is defined with respect to the same augmentation family used in pre-training; the authors note in the Conclusion that the choice of augmentation for time-series contrastive learning is an open question, so transferability of the metric across augmentation distributions is an unresolved load-bearing assumption.","section":"Sec. 3.3, Eq. (3)"}],"minor_comments":[{"comment":"The sentence 'For each pair (X0, Xnew0' is missing a closing parenthesis; it should read 'For each pair (X0, Xnew0)'.","section":"Sec. 4.2"},{"comment":"The dataset name 'AllGestureWiimo-teX' is misspelled; it should be 'AllGestureWiimoteX', and the labels in Figures 5 and 6 should be checked for consistency (e.g., 'WormsTwoClass' vs 'WormsTwoClass').","section":"Appendix B"},{"comment":"The Lag-Llama reference appears twice with identical content (Rasul et al., 2023a and 2023b); if these are the same paper they should be merged, and if they are different they should be distinguished by title or venue.","section":"References"},{"comment":"The notation for Acon is inconsistent: Eq. (3) writes A(X')con (X0), while Section 4.1 writes A(X0)con (X(r%)0). The superscript/subscript convention for the evaluation set and the pre-training set should be defined once and used consistently.","section":"Sec. 3.3 / Sec. 4.1"},{"comment":"The x-axis labels mix text and math ('Contrastive acc improvement, Acon (X0, X(new)0)'); use a defined symbol such as ΔAcon throughout for readability.","section":"Figures 5-6"},{"comment":"The paper does not mention whether code or exact UCR data splits will be released; for a metric paper, releasing the pre-training and evaluation code would greatly improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-length study with an interesting and potentially useful idea. The Section 4.1 evidence is encouraging, but the Section 4.2 analysis, which is the only direct support for the dataset-selection claim, needs substantial statistical work. If the mixed results persist after a proper run-level or mixed-effects analysis, the authors should narrow the claim from 'can serve as a criterion to search' to 'can provide a signal in some settings' and discuss the failure modes (e.g., EOGVerticalSignal). I would not reject the paper at this stage, but the current version overstates what the experiments establish. I would also encourage the authors to release code and exact data splits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is simple and worth knowing: contrastive accuracy, the mutual-nearest-neighbor rate of two augmented views, as a label-free proxy for downstream accuracy of a time series foundation model. The framing is good — connecting the alignment/uniformity view of contrastive learning to unlabeled performance estimation is a natural move that I don't think anyone has made for time series foundation models yet. The practical motivation is real: if this works, practitioners get a cheap way to screen candidate pre-training data without fine-tuning on labeled downstream tasks.\n\nThe paper does something genuinely useful in Section 4.1. Across four UCR datasets, subsampling the pre-training set and measuring CA gives correlations with downstream accuracy between 0.62 and 0.99. Those are strong, and the metric is cheap to compute. Credit where due: the idea is clear, the experiments are reproducible in principle, and the authors honestly flag in the conclusion that augmentation choice for time series contrastive learning is open.\n\nThe soft spots are concentrated in Section 4.2, which is the only experiment that actually tests the \"quality\" interpretation by swapping dataset identity rather than subsample size. In the fix-X0 panels (Figure 5), only 6 of 12 datasets show significant positive correlation; EOGVerticalSignal is negative (rho = -0.07), and five others are non-significant. The main text illustrates with AllGestureWiimoteX, one of the strongest panels, and hides the full distribution in the appendix. The vary-X0 panels (Figure 6) are stronger, but that direction is less practically relevant. Also, the Section 4.1 correlations are confounded with the subsample ratio r, so they don't establish ranking of datasets by quality. The p-value reporting is ambiguous — computed on 10 or 11 averaged points, not on the 50 individual runs — and there is no code, no comparison to simpler baselines like average pairwise distance or a pure uniformity score.\n\nThese are real but addressable problems. The central claim — that CA correlates with downstream accuracy — is plausible and partially supported; the abstract's stronger claim that CA \"can serve as a criterion to search for time series datasets\" is not established by the reported evidence. The paper deserves a serious referee, not a desk reject. For a full venue, it needs the complete 12-panel results in the main text, statistics over runs, code, and baseline comparisons. For a workshop paper, it's acceptable as a preliminary study.\n\nMy recommendation: send it to peer review, but expect revision or a transfer to a workshop track. I'd bring it to a reading group interested in time series foundation models or unsupervised performance estimation.","headline":"Useful label-free proxy idea, but the paper's own improvement-prediction experiment is too mixed to carry the data-selection claim; needs revision, not rejection.","tokens_in":9958,"tokens_out":1402,"would_cite":true,"duration_ms":15516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A label-free measure of embedding scatter predicts how well time-series pre-training data will improve a foundation model.","keywords":["contrastive accuracy","pre-training data selection","time series foundation models","contrastive learning","label-free evaluation","embedding uniformity","UCR time series archive"],"falsifier":"Take a diverse pool of candidate time-series datasets, compute contrastive accuracy for each under the fixed pre-trained model, and then measure the actual downstream accuracy after adding each dataset to pre-training; if the rank correlation between Acon and accuracy is close to zero or negative on this pool, the paper's central claim would be refuted. A sharper test would find two datasets with similarly high Acon where adding one improves downstream accuracy while adding the other degrades it, breaking the claimed monotonic relationship.","tokens_in":8980,"feed_emoji":"📈","tokens_out":5559,"duration_ms":52919,"temperature":0.7,"pith_summary":"This paper tries to answer a practical question: when building a time series foundation model by contrastive pre-training, how can you tell whether a candidate unlabeled dataset will improve the model, without running the full downstream evaluation? The authors propose contrastive accuracy (Acon), a label-free metric computed from the model's own embeddings: it measures how scattered the embeddings of the candidate data are, by checking whether two augmented views of the same example become mutual nearest neighbors within a batch. They report that Acon correlates strongly with the model's averaged accuracy on a collection of 127 UCR classification tasks when the pre-training set is subsampled, and that changes in Acon predict the performance change from adding a new dataset to pre-training. If these correlations hold in broader settings, Acon would give a cheap criterion for selecting and expanding pre-training corpora for time series foundation models without needing labels or repeated fine-tuning.","feed_headline":"Label-free metric predicts pre-training data quality","feed_subtitle":"Contrastive accuracy tracks downstream accuracy across 127 UCR tasks, flagging which unlabeled datasets help pre-training.","key_machinery":"The load-bearing object is the contrastive accuracy Acon, computed as the expectation over random augmentations of the fraction of samples whose augmented embedding has its own other augmented embedding as the nearest neighbor in the batch, using cosine similarity. It operationalizes the uniformity property of contrastive representations (following Wang and Isola, 2020): instead of estimating the distribution on the unit hypersphere, it counts self-nearest-neighbor matches, which is cheap to compute from batches. The metric is defined relative to a fixed pre-trained model and a fixed augmentation family (RandomCropResize with crop ratio 0.7 to 0.8), so it is a model-dependent, label-free proxy rather than a pure property of the data alone.","core_discovery":"The central claim is that the quality of a pre-training dataset for a time series foundation model can be evaluated without labels by measuring how well its embeddings satisfy the uniformity property of contrastive learning. Concretely, for a candidate dataset X', the contrastive accuracy Acon is the expected fraction of samples whose two random augmentations produce embeddings that are the nearest neighbor of each other within the batch; a high value means the representation space is well spread, which the paper argues preserves information and supports downstream discrimination. The paper shows that when a fraction r% of a large UCR dataset is used for pre-training, Acon computed on the subsample tracks both training and test accuracy on the remaining 127 datasets (correlations up to rho = 0.99 in the ElectricDevices case), and that the difference in Acon between two pre-training sets correlates with the difference in downstream performance (rho values around 0.74 to 0.75 for the shown AllGestureWiimoteX case, with a range across 12 datasets). The authors frame this as a way to guide dataset collection and selection for time series foundation models.","pith_inferences":["The metric inherits the quality of the augmentation family; the authors flag the choice of augmentations for time series as an open question, so Acon's rankings may shift across augmentation choices and should be tested for stability.","Acon measures uniformity but not alignment or downstream class structure, so it may complement rather than replace other label-free performance estimators such as projection-norm or confidence-based methods from the out-of-distribution literature.","The correlations are reported on UCR; testing on larger and more heterogeneous pre-training pools, including forecasting or multivariate series, would calibrate how the relationship holds as domain diversity grows.","Because the uniformity principle is general, the same metric could be applied to contrastive foundation models in image or speech domains, offering a natural test bed beyond time series."],"forward_implications":["Dataset curation for time series foundation models can be done by ranking candidate unlabeled datasets on contrastive accuracy and picking those with high values, without supervised evaluation.","The subsampling experiments suggest that Acon can indicate when enough examples have been included from a given dataset, guiding the scale of pre-training corpora.","The improvement-prediction results suggest that Acon differences can forecast whether adding a particular dataset to pre-training will help, enabling greedy data expansion.","Because Acon requires only the model and unlabeled candidate data, it can be used repeatedly during training as a cheap monitor of representation quality."],"supporting_citations":[{"why":"Supplies the alignment-uniformity principle that motivates contrastive accuracy as a uniformity measure.","marker":"(Wang and Isola, 2020)"},{"why":"Provides the UCR time series archive used both for pre-training and for the downstream evaluation suite.","marker":"(Dau et al., 2019)"},{"why":"Introduces the InfoNCE contrastive loss that the paper minimizes during pre-training.","marker":"(Oord et al., 2018)"},{"why":"Together with Oord et al., grounds the contrastive learning framework used to train the foundation model.","marker":"(He et al., 2020)"},{"why":"Provides the ViT-style patch-based architecture for time series that the paper adapts for its backbone.","marker":"(Nie et al., 2023)"},{"why":"Offers a directly relevant time-series pre-training model with a transformer architecture used as a reference for the implementation.","marker":"(Lin et al., 2024)"},{"why":"Defines the framework of unlabeled performance estimation that the paper positions itself against.","marker":"(Donmez et al., 2010)"}],"fun_headline_variants":["No labels needed: contrastive metric scores data quality","Data quality without labels? Measure contrastive accuracy","Contrastive accuracy: the label-free data quality gauge","Unlabeled data quality score predicts downstream accuracy","Skip labels: use contrastive accuracy to pick pre-training data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The metric's predictive power rests on the assumption that a candidate dataset's embedding uniformity, measured with one fixed augmentation family, accurately reflects the model's downstream classification accuracy on other datasets across distribution shifts.","fun_headline_variants_meta":{"raw":{"variants":["No labels needed: contrastive metric scores data quality","Data quality without labels? Measure contrastive accuracy","Contrastive accuracy: the label-free data quality gauge","Unlabeled data quality score predicts downstream accuracy","Skip labels: use contrastive accuracy to pick pre-training data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1514,"prompt_tokens":888,"completion_tokens":626,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":564}},"tokens_in":504,"tokens_out":626,"duration_ms":5408,"temperature":1.0,"reasoning_tokens":564,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:45:01.441100+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a diverse pool of candidate time-series datasets, compute contrastive accuracy for each under the fixed pre-trained model, and then measure the actual downstream accuracy after adding each dataset to pre-training; if the rank correlation between Acon and accuracy is close to zero or negative on this pool, the paper's central claim would be refuted. A sharper test would find two datasets with similarly high Acon where adding one improves downstream accuracy while adding the other degrades it, breaking the claimed monotonic relationship.","supporting_citations":[{"cited_title":"and Isola, P","cited_arxiv_id":null,"evidence_quote":"Supplies the alignment-uniformity principle that motivates contrastive accuracy as a uniformity measure."},{"cited_title":"A., Bagnall, A., Kamgar, K., Yeh, C.-C","cited_arxiv_id":null,"evidence_quote":"Provides the UCR time series archive used both for pre-training and for the downstream evaluation suite."},{"cited_title":"Nguyen, N., Sinthong, P., and Kalagnanam, J","cited_arxiv_id":null,"evidence_quote":"Provides the ViT-style patch-based architecture for time series that the paper adapts for its backbone."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers a directly relevant time-series pre-training model with a transformer architecture used as a reference for the implementation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the framework of unlabeled performance estimation that the paper positions itself against."}],"review_version":1}