{"id":"7293c77c-2aed-4209-8139-96397b3b9392","arxiv_id":"2504.20099","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Fine-tuning MOMENT time series foundation models reduces reconstruction loss but does not visually improve the interpretability of their latent space projections in the DeepVATS visual analytics environment.","lead":"The authors integrated MOMENT, a time series foundation model, into the DeepVATS visual analytics tool and compared zero-shot and fine-tuned embeddings on five datasets. They found that fine-tuning reduces reconstruction loss but does not clearly improve how interpretable the embedding projections look by eye.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conclusion that fine-tuning does not improve interpretable embeddings rests on unvalidated 2D projections; without quantitative or seed-robust checks, the negative visual result may be a projection artifact.","rationale":"The reader's conditional verdict and weakest-assumption analysis are sound: the work is an exploratory integration with a clear loss analysis, but the interpretability conclusion lacks a quantitative or projection-validated basis. My stress-test converges on the same load-bearing point—the visual comparison is the entire evidence for the headline negative result, and UMAP/PCA are neither faithful nor stable enough to support that inference without control. This does not move the verdict: CONDITIONAL remains appropriate, since the central claim should be treated as a hypothesis requiring quantitative evaluation rather than an established finding. The paper is honest about this in its limitations and future-work section, and the released code makes the proposed check feasible.","tokens_in":18149,"tokens_out":7946,"duration_ms":86663,"concrete_test":"On the released DeepVATS-MOMENT checkpoints, compute label-aware cluster metrics (e.g., silhouette score, Davies-Bouldin index, and adjusted Rand index against known S1/S2/S3/M-Toy/Kohl labels) directly on the raw embedding vectors before and after fine-tuning, without any projection. Then rerun UMAP with at least 10 random seeds and compare PCA, t-SNE, and PacMAP projections of the same embeddings. If any metric or alternative projection reveals improved fine-tuned structure, the visual no-change conclusion is a projection artifact; if all remain unchanged, the conclusion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that reconstruction-loss reduction after fine-tuning does not translate into visually interpretable MOMENT embeddings (Section IV)—is supported only by visual similarity and difference of scatter plots (Figs. 10–22). This assumes the UMAP-then-PCA pipeline inherited from DeepVATS faithfully and stably represents MOMENT's high-dimensional embedding geometry. The paper reports no cluster-quality metric on the raw embeddings, no label-aware separation score, no inter-rater reliability, and no control for UMAP stochasticity; the projection hyperparameters were selected for MTSAE (Section III.C), not for MOMENT. Additionally, the reported loss improvement is a masked reconstruction loss; the paper never measures how much the embedding vectors themselves changed, so the comparison conflates reconstruction behavior with encoder geometry. A visually unchanged scatter plot can mean unchanged embeddings, but it can also mean the projection is insensitive to the actual changes or that random-seed and hyperparameter variation dominates. The paper's own suggestion to try alternative projection methods (Section IV) concedes exactly this ambiguity. Until the projection is validated, the negative visual finding is not distinguishable from a projection artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper integrates the MOMENT family of time series foundation models into the DeepVATS visual analytics framework and studies whether zero-shot and fine-tuned MOMENT embeddings yield visually interpretable projections for segmentation, anomaly, pattern, and trend detection. The authors evaluate MOMENT-small/base/large on five datasets (S1, S2, S3, M-Toy, Kohl's), fine-tune the models with a masked-reconstruction objective, select hyperparameters via feature-importance analysis on Kohl's, and compare loss improvements with visual inspection of UMAP-then-PCA scatter plots. They report substantial loss reductions after fine-tuning (up to ~22%) but little visible improvement in embedding interpretability, and conclude that quantitative improvements are not necessarily linked to embedding precision. The paper's stated contribution is the first integration of a foundation model into a visual analytics tool, with a wrapper for fine-tuning arbitrary torch-based models.","tokens_in":18327,"tokens_out":6157,"duration_ms":55077,"significance":"If the negative visual result were backed by quantitative evidence, the paper would be a useful cautionary case study for the visual analytics and time-series foundation model communities: it would show that reconstruction-loss gains do not automatically translate into human-interpretable projections and would motivate alternative projection/loss/preprocessing choices. The paper has concrete strengths: code is promised via a GitHub footnote, the study covers three model sizes and five datasets, the statistical analysis uses multiple feature-importance estimators (SelectKBest, Random Forest, SHAP, and correlation), and the integration wrapper is a reusable engineering contribution. However, the central conclusion rests on the authors' subjective reading of scatter plots; no quantitative interpretability metric, no inter-rater reliability, no ablation of projection choice, and no repeated-run variance are reported. Because the projection pipeline (UMAP followed by PCA) is inherited from the earlier MTSAE-based DeepVATS study and not validated for MOMENT embeddings, the stress-test concern lands: the negative finding is currently indistinguishable from a projection artifact.","major_comments":[{"comment":"The central claim that fine-tuning does not improve embedding interpretability is based solely on the authors' visual comparison of scatter plots. No quantitative interpretability metric (e.g., cluster purity, silhouette score, label-aware separation, or projection trustworthiness) is computed on the raw MOMENT embeddings; no inter-rater reliability is reported; and UMAP stochasticity is not controlled by repeated runs or fixed seeds. The paper's own Section IV recommends testing alternative projection methods (e.g., PCA without UMAP and t-SNE), which concedes that the current projection may obscure genuine structure. To make the negative visual result load-bearing, the authors should report embedding-space quality metrics computed before projection, repeat UMAP with several seeds, and compare at least one alternative projection method. Without this, the conclusion that fine-tuning leaves the embeddings visually unchanged is not distinguishable from a projection artifact.","section":"§III.D–IV, Figs. 10–22"},{"comment":"The hyperparameter selection procedure is not sufficiently justified or robust. The grid is first reduced from 72 to 18 cases for computational reasons, and the final parameters in Table 6 differ from the 'best' values in Table 5 (e.g., MOMENT-base masked_percent changes from 25 to 15 and dataset_percent from 15 to 25; MOMENT-large best_epoch changes from 17 to 10) without explanation of the cost-to-improvement trade-off used. These hyperparameters are tuned on the Kohl's time series and then applied to all other datasets (S1, S2, S3, M-Toy), yet no justification is given for transferring them across datasets with different structure. In addition, the reported loss improvements are single-run values without standard errors or repeated seeds, so the 22% / 10% / 4% improvements cannot be assessed for stability. The authors should report variance across runs and justify the final parameter choices, or the fine-tuning comparison is not reliably interpretable.","section":"§III.C, Tables 5–6"},{"comment":"The paper conflates reconstruction loss with embedding geometry. The improvement metric is the masked reconstruction loss improvement, defined as (loss_first − loss_final)/loss_first × 100 in §III.C. No measure of how much the embedding vectors themselves changed (e.g., mean cosine or euclidean displacement, nearest-neighbor overlap) is reported, and the phrase 'embeddings precision' in the Introduction and Conclusions is never defined. Consequently, the conclusion that 'quantitative improvements are not necessarily linked to the embeddings precision' is not supported by the measurements presented; a visually unchanged projection could reflect unchanged embeddings, projection insensitivity, or dominant seed and hyperparameter variation. The authors should either define and measure embedding-space change directly or soften the claim to refer specifically to reconstruction loss.","section":"§III.C–IV, RQ1"}],"minor_comments":[{"comment":"The manuscript needs thorough language editing; there are numerous typos and grammatical errors (e.g., 'supose', 'wether', 'apportation', 'intrinsecal', 'diferent', 'descripted', 'freexing', 'difficultying').","section":"Throughout"},{"comment":"The caption text says these figures show 'MOMENT-small for S1', but Section III.E and the surrounding analysis refer to MOMENT-base clusters; the caption should be corrected to MOMENT-base.","section":"Appendix B.1, Figs. 40–42"},{"comment":"The captions labeled 'From top to botton, the linear correlation matrices for MOMENT-small/base/large' do not match the surrounding text, which describes these figures as the frequency of the best epoch; the captions should be corrected.","section":"Appendix A, Figs. 29–31"},{"comment":"The column header 'Best %' in Table 4 is inconsistent with the 'KBest %' header used in Tables 2 and 3; the table also should state how the percentages are normalized across the four feature-importance estimators.","section":"Table 4"},{"comment":"The summary table is numbered Table 12 although no Table 11 appears in the text; renumber the tables sequentially.","section":"Tables 11–12"},{"comment":"The text says the full grid has 72 cases and then reduces it to 18 cases due to computational cost; please state exactly which of the 72 configurations were actually run and whether the reduction was a random subsample or a systematic restriction.","section":"§III.C"},{"comment":"The claim that foundation models 'supose a big reduction in execution time' is not supported by any runtime measurements in the paper; if kept, it should be backed by measured training or inference times or removed.","section":"§IV"},{"comment":"The claim of being 'the first integration of foundation models into visual analytics tools' is stated without a systematic comparison to existing work; please soften the novelty claim or provide evidence from a broader literature search.","section":"§I"}],"recommendation":"major_revision","confidential_remarks":"The paper is a case study with a largely negative result; it may be better positioned as an application note than as a foundational claim. The editors should verify the 'first integration' novelty claim, as it is stated without a comprehensive literature survey. The central issue is the lack of quantitative validation of the visual interpretability claim; this is fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper does something genuinely new by wiring MOMENT into DeepVATS and then reports an honest negative: fine-tuning cuts masked reconstruction loss (up to ~22%) but the 2D projections of the embeddings look essentially unchanged. That is a cautionary data point for anyone building visual analytics on top of time series foundation models. It deserves a serious referee, but it needs quantitative support before the conclusion can be trusted.\n\nThe integration itself is the strongest part. They provide a wrapper so any torch model can be dropped into DeepVATS, and they've shared code and data on GitHub. The hyperparameter study is systematic: 18 configurations per model, with feature importance analysis. Their finding that masked_percent and best_epoch dominate dataset_percent and n_windows is clear and reproducible. The comparison to MTSAE is also useful: MOMENT starts with much lower MSE, so the headroom is small.\n\nThe soft spot is the central decoupling claim. It rests on eyeballing scatter plots (Figs. 10–22). There is no cluster quality metric on the raw embeddings, no label-aware separation score, no inter-rater reliability, and no seed repetition. More importantly, the UMAP→PCA pipeline was inherited from DeepVATS, where it was tuned for MTSAE, not MOMENT. So the null visual result could be a projection artifact. The paper itself half-admits this in Section IV when it suggests trying other projection methods. Also, loss is masked reconstruction loss; the authors never measure how much the embedding vectors actually changed, so 'no visual improvement' doesn't yet mean 'no embedding change.' The loss numbers appear to be single runs, so the 22% improvement has no error bar.\n\nThese are real limitations, but they are not disqualifying for an exploratory paper. The authors are appropriately cautious in the abstract, and they present the result as a question for further work rather than a hard claim. The prose has typos and some overreach in the intro ('significantly change a model's internal knowledge' goes beyond the data), but the core empirical observation is honest and potentially useful.\n\nMy recommendation: send to peer review. The reviewers should ask for a quantitative interpretability analysis (silhouette or similar on raw embeddings, plus seed/run variability and an alternative projection or two). With those additions, this could become a solid reference point for TSFM interpretability work. I'd likely cite it as a cautionary example.","headline":"A useful, honest negative result: fine-tuning MOMENT reduces reconstruction loss but doesn't visibly improve its embedding projections, though the evidence is only visual.","tokens_in":18894,"tokens_out":3656,"would_cite":true,"duration_ms":32363,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a time-series foundation model lowers reconstruction loss, but its latent-space projections stay just as hard to read.","keywords":["time series foundation models","latent space interpretability","visual analytics","MOMENT","fine-tuning","embedding visualization","UMAP","reconstruction loss"],"falsifier":"Compute cluster-separation metrics, such as silhouette score or cluster purity, on the raw embeddings before any projection for zero-shot versus fine-tuned MOMENT models on the S1 segmentation dataset; if these metrics improve substantially after fine-tuning, the claim that loss gains do not translate into embedding precision would need to be revised.","tokens_in":17928,"feed_emoji":"📉","tokens_out":8221,"duration_ms":75702,"temperature":0.7,"pith_summary":"The paper asks whether the quantitative gains from fine-tuning a time-series foundation model show up as more readable latent-space visualizations. Using the MOMENT family of transformer-based models and five datasets, it compares zero-shot embeddings with embeddings after lightweight masked-imputation fine-tuning, projecting both through the same UMAP-then-PCA pipeline used by the DeepVATS visual-analytics tool. Fine-tuning cut reconstruction loss by up to about 22 percent, yet the projected clusters changed little and stayed highly intertwined: segments, anomalies, and trends remained hard to isolate. The authors conclude that loss improvement and embedding interpretability are not reliably linked, and that even minimal fine-tuning can substantially alter internal knowledge, which supports using small, fast foundation models as backbones for interactive visual analytics.","feed_headline":"Loss drops 22 percent, yet time-series embedding plots stay tangled","feed_subtitle":"Reconstruction loss and readable embeddings can move independently in MOMENT time-series models.","key_machinery":"The mechanism that carries the argument is the pairing of MOMENT's patch-based transformer embeddings with a fixed visual-analysis protocol: time series are cut into windows, embedded by the model, then projected to two dimensions with UMAP followed by PCA for inspection in DeepVATS. The comparison of loss improvement, defined as the percent reduction in masked mean-squared error before versus after fine-tuning, against the visual structure of these projections is what allows the paper to separate quantitative adaptation from interpretability. Fine-tuning itself is done as a masked-imputation task with four tunable parameters: the percentage of data used for training, the validation percentage, the mask percentage, and the number of Fourier-selected window lengths.","core_discovery":"The central claim is that for the MOMENT family of multivariate time-series foundation models, reductions in reconstruction loss after task-specific fine-tuning do not translate into clearer latent-space projections. Across segmentation, anomaly-detection, and trend datasets (S1, S2, S3, M-Toy, and Kohl's), both zero-shot and fine-tuned versions produced embedding plots with interlinked clusters; the same segments, anomalies, and trends were equally hard to isolate before and after training. MOMENT-small showed the largest loss improvements (around 22 percent), MOMENT-base the clearest partial segmentation of the S1 series, and MOMENT-large the most defined clusters for anomaly detection, yet no model version showed a visually interpretable separation of trends, and fine-tuning rarely changed the projection beyond rotations or subtle rearrangements. The paper therefore argues that the apparent understanding gained by the model, as measured by loss, is not reflected in projection-based interpretability, and that the MOMENT embeddings may need alternative projection methods, loss functions, or preprocessing to support visual analytics.","pith_inferences":["Going beyond the paper, the negative visual result may be an artifact of the projection: UMAP followed by PCA with fixed parameters can obscure structure that a different projection, such as t-SNE or PacMAP, might reveal.","A natural testable extension is to measure cluster purity or silhouette scores on the raw embeddings before projection; if those metrics improve after fine-tuning while the projected plots do not, the bottleneck is visualization, not the model's internal geometry.","If the projection-artifact explanation holds, the same experimental design could serve as a reusable benchmark for future time-series foundation models, provided that projection-free interpretability metrics are added alongside the visual inspection."],"forward_implications":["Reconstruction loss should not be used as a proxy for embedding clarity in visual-analytics applications; a model can appear to improve numerically while its projected clusters stay just as tangled.","For interactive visual analytics, the smallest model version is the most attractive: it had the largest loss reduction, similar interpretability to the larger versions, and lower computation and memory cost.","Fine-tuning with a small fraction of the data (15 to 20 percent) and a modest number of epochs is enough to change model behavior, so heavy training is not required for adaptation.","Time-series foundation models can replace task-specific autoencoders in visual-analytics pipelines, cutting waiting time while retaining pattern detection, but they do not yet provide clear segment and trend structure by default.","Improving interpretability will require modifying the pipeline itself, for example by trying alternative projection techniques, loss functions such as soft-DTW, or data preprocessing, rather than simply fine-tuning longer."],"supporting_citations":[{"why":"Supplies the DeepVATS visual-analytics pipeline, the datasets (S1, S2, S3, M-Toy, Kohl's), and the UMAP/PCA projection settings used for the interpretability comparisons.","marker":"[5]"},{"why":"Introduces the MOMENT family of pre-trained time-series transformers and the earlier claim that their embeddings project interpretably, which this paper tests under fine-tuning.","marker":"[10]"},{"why":"Defines UMAP, the first stage of the projection pipeline whose visual output grounds the interpretability judgments.","marker":"[30]"},{"why":"Introduces the TimeInception architecture that forms the backbone of the MTSAE baseline whose latent-space interpretability is the comparison standard.","marker":"[37]"},{"why":"Supplies the segmentation toolkit function used to select dominant window sizes for the fine-tuning experiments.","marker":"[40]"},{"why":"Soft-DTW is proposed as an alternative loss that could yield more refined embeddings, indicating where the paper locates the fix for the negative result.","marker":"[41]"}],"fun_headline_variants":["Loss drops, but time-series embeddings stay tangled","Fine-tuning cuts loss, not embedding clarity in MOMENT models","MOMENT's hidden spaces resist visual decoding despite better loss","Lower loss, same tangled plots: MOMENT's interpretability gap","Time-series AI: better predictions, still opaque latent views"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The negative conclusion assumes that the UMAP-then-PCA projections used in the DeepVATS pipeline faithfully expose the structure of the high-dimensional embeddings; if a different projection would reveal clear clusters, the visual result would be an artifact of the projection rather than of the embeddings.","fun_headline_variants_meta":{"raw":{"variants":["Loss drops, but time-series embeddings stay tangled","Fine-tuning cuts loss, not embedding clarity in MOMENT models","MOMENT's hidden spaces resist visual decoding despite better loss","Lower loss, same tangled plots: MOMENT's interpretability gap","Time-series AI: better predictions, still opaque latent views"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1569,"prompt_tokens":938,"completion_tokens":631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":546}},"tokens_in":554,"tokens_out":631,"duration_ms":6106,"temperature":1.0,"reasoning_tokens":546,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:05:56.736553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute cluster-separation metrics, such as silhouette score or cluster purity, on the raw embeddings before any projection for zero-shot versus fine-tuned MOMENT models on the S1 segmentation dataset; if these metrics improve substantially after fine-tuning, the claim that loss gains do not translate into embedding precision would need to be revised.","supporting_citations":[{"cited_title":"Inceptiontime: Findingalexnetfor time series classification,","cited_arxiv_id":null,"evidence_quote":"Introduces the TimeInception architecture that forms the backbone of the MTSAE baseline whose latent-space interpretability is the comparison standard."},{"cited_title":"aeon: a python toolkitforlearningfromtimeseries,","cited_arxiv_id":null,"evidence_quote":"Supplies the segmentation toolkit function used to select dominant window sizes for the fine-tuning experiments."},{"cited_title":"Soft-dtw: a differentiable loss function for time-series,","cited_arxiv_id":null,"evidence_quote":"Soft-DTW is proposed as an alternative loss that could yield more refined embeddings, indicating where the paper locates the fix for the negative result."}],"review_version":1}