{"id":"9b573193-8b7f-4988-83eb-1511b7cf9ce4","arxiv_id":"2505.15072","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MoTime provides a large multimodal forecasting benchmark and shows that external text or images can improve forecasts in some datasets, especially cold-start and sparse settings, though gains are inconsistent.","lead":"MoTime is a new collection of eight public datasets that pair time series of item popularity, views, and sales with text, images, and metadata, spanning roughly 869,000 series. The paper tests when these extra signals help forecasting, including cold-start tasks with no history, and finds gains vary by dataset and horizon.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scraped external text (Wikipedia summaries for WikiPeople, overviews for MovieLens) is not timestamp-audited; post-forecast content could leak into cold-start inputs and inflate reported modality gains.","rationale":"The reader's weakest_assumption matches my concern exactly: the cold-start claim hinges on text that may contain information from after the forecast period. The dataset's infrastructure value is independent of this bug—the archives still provide aligned series and modalities, and the leakage is fixable by re-scraping with historical revisions or timestamp-filtering text. The reported cold-start results, however, are not trustworthy until this audit is done. Since the reader's CONDITIONAL verdict already captures this, my stress-test confirms the verdict rather than changing it.","tokens_in":24834,"tokens_out":4930,"duration_ms":46038,"concrete_test":"For WikiPeople, fetch the 2017-09-01 (or forecast-date) revision of each article summary via Wikipedia's API, and re-run the cold-start retrieval + GPT forecast with those historical summaries in place of the current ones; then recompute Figure 2. If the GPT advantage over the average baseline shrinks by more than, say, 10% of the originally reported gap, the leakage concern is confirmed. As a supplementary check, count how many of the 30 sampled WikiPeople series have summaries mentioning events after the forecast period.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states that for MovieLens and WikiPeople, text is 'obtained by crawling movie metadata or Wikipedia summaries' and aligned via IDs, but no timestamp check is described. Section 4.2's cold-start protocol feeds the target's textual description into retrieval and GPT generation while removing all but the first valid observation. For WikiPeople, whose series end in mid-2017 (Kaggle Web Traffic competition), current Wikipedia summaries (scraped ~2025) routinely contain post-2017 facts such as deaths, elections, or career events. This gives the cold-start model future knowledge relative to the forecast point, violating the no-history premise. A person's death is a strong predictor of page-view spikes; knowing it occurred after the forecast window can bias predicted levels even when forecasting pre-event dates, and the 'average of retrieved series' baseline does not have access to this leaked semantic knowledge, so the reported GPT-vs-average gap in Figure 2 may be inflated. AmazonReview similarly includes aggregate rating metadata computed over the full 2013-2023 period, which post-dates cold-start forecast windows. Because the central empirical claim—that external modalities improve cold-start forecasting—rests on exactly these text-conditioned inputs, this is the load-bearing risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MoTime, a suite of eight multimodal time-series forecasting datasets constructed by re-purposing recommender-system and web-popularity data. Each dataset pairs time series with static text, image, and/or metadata modalities, and the suite is released publicly. The authors define two evaluation scenarios: varying-history forecasting, where short and long historical windows are contrasted, and cold-start forecasting, where no target history is available and predictions are made from textual descriptions, retrieved similar series, and GPT-4o-mini generation. Experiments with adapted PatchTST and WPMixer variants, plus LLM-based cold-start forecasts, are used to argue that external modalities can improve forecasting in both scenarios, with gains that vary by dataset, sparsity, and horizon.","tokens_in":25065,"tokens_out":4415,"duration_ms":38939,"significance":"If the empirical claims hold, MoTime would be a valuable infrastructure contribution: it is substantially larger than existing multimodal time-series benchmarks (869,466 series across eight domains), includes static entity-level text/image/metadata alignments rather than only dynamic event streams, and explicitly operationalizes cold-start forecasting, which is rarely benchmarked. The paper is also transparent about many construction details and limitations, and the data release is a concrete asset for the community. However, the central claim that external modalities improve cold-start forecasting is currently supported by experiments that lack a leakage audit and statistical grounding, so the significance is conditional on those issues being resolved.","major_comments":[{"comment":"The cold-start modality-utility claim is exposed to temporal leakage from externally scraped text. Section 3.2 states that for MovieLens and WikiPeople text is obtained by crawling movie metadata or Wikipedia summaries and linked by IDs, but no timestamp check is described. WikiPeople series end in mid-2017 (the Kaggle competition), while Wikipedia summaries scraped circa 2025 can contain post-2017 facts such as deaths, elections, or later career events. The cold-start protocol in Section 4.2 removes all but the first valid observation but feeds the target's current text description into retrieval and GPT generation; the retrieved-series averaging baseline does not receive this semantic knowledge. This violates the no-history premise and can inflate the GPT-versus-average gap in Figure 2. AmazonReview has a parallel issue: aggregate rating metadata computed over the full 2013--2023 period post-dates the cold-start forecast points. The authors should either use timestamped text snapshots (e.g., Wikipedia revision dates, aggregated metadata computed only up to each prediction point) or demonstrate empirically that the reported cold-start results are unchanged under a strict leakage audit.","section":"§3.2, §4.2, Fig. 2"},{"comment":"All headline comparisons are reported without confidence intervals, standard deviations across seeds, or significance tests. Many differences are at the third decimal (e.g., Table 3, Amazon horizon 1: MultiPatchTST 0.373 vs DLinear 0.373, with bold/underline assigned by full precision). The large number of series per dataset could make standard errors very small, but that is precisely why significance tests, effect-size reporting, or seed variance are needed to distinguish real gains from noise. Moreover, comparing PatchTST with MultiPatchTST and WPMixer with MultiWPMixer changes both the architecture and the input modality; a genuine modality-utility estimate requires a controlled ablation with the same architecture and the text/image input dropped, or a statistical interaction test.","section":"§5.1, Tables 3 and 15"},{"comment":"The short-series benefit claim is confounded by training imbalance. Section 4.1 trains jointly on a 1:1 split of short and long series with no upsampling, and Section 6 acknowledges that short series may contribute less to gradient updates. Since the abstract claims 'particularly strong benefits for short series in some datasets,' the observation that MovieLens short series improve with MultiWPMixer (Section 5.2.1) could be an artifact of underfitting short series rather than a modality effect. The authors should rebalance the training data (e.g., oversample short windows or add a weighting term) and rerun, or at least report a sensitivity analysis varying the short/long ratio and the horizon thresholds listed in Section 5.1.","section":"§4.1, §6, Table 15"},{"comment":"Cold-start evaluation uses a fixed random sample of 30 series per dataset and a single top-4 retrieval setting. With sparse, bursty series (e.g., PixelRec and Tianchi), single-split results on such a small sample are unstable, especially when WRMSPE denominators are near zero. The paper reports no variance across sampling seeds and no sensitivity to the retrieval top-k, the cold-start sample size, or the PixelRec density-0.4 and MovieLens 1,000-series filters introduced in Section 5.1. I therefore do not trust the specific magnitudes in Figure 2, even if the qualitative direction may be right.","section":"§5.2.2, §5.1"}],"minor_comments":[{"comment":"The abstract says external modalities give 'particularly strong benefits for short series in some datasets,' but the main text concludes the short-series hypothesis holds only in specific cases and that performance trends are largely consistent across short and long subsets; the abstract should be softened to match this nuance.","section":"Abstract, §5.2.1"},{"comment":"VISUELLE is listed in Table 2 and Section 3.1 but is not processed or experimented on; the main text should state clearly whether VISUELLE is included in the released suite and, if so, why it appears in the statistics table. Also, Section 3.1 says 'The time series is item sales' for VISUELLE, which appears inconsistent with the engagement-statistics description.","section":"§3.1, Table 2, §3.2"},{"comment":"The News dataset evaluation uses a filtered subset (topic 'obama', platform Facebook), which is described only in Appendix 8.1.6; this filtering decision is important for reproducibility and should be stated in the main text, ideally in Section 3.1 or Section 5.1.","section":"§3.1, §5.1, §8.1.6"},{"comment":"There are several presentation issues: 'obervations' in Section 3.3, inconsistent spellings 'Movielens' vs 'MovieLens', unformatted WikiPeople RMSE values such as '21977' in Table 3, and the placeholder 'gragality' in the prompt template of Table 13.","section":"§3.3, Table 3, Table 13"},{"comment":"The AmazonReview per-category text statistics show identical minimum, maximum, mean, and median values for many categories (e.g., char_len_max=3938 for many categories), which suggests the statistics describe a single long concatenated metadata string; please clarify how these text fields are constructed and whether truncation is applied before embedding.","section":"Tables 7--10"},{"comment":"Figure 2 reports cold-start results only as bars without numeric values or error bars; please add an appendix table with the exact RMSE/WRMSPE values and repeated-seed variation so readers can assess the magnitude of the GPT-versus-average differences.","section":"Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"The dataset itself is a useful resource and the construction is described in commendable detail. The main risk is that the cold-start empirical claims rest on text that may contain post-forecast information, and the statistical evidence for modality utility is thin. A leakage audit and a modest amount of additional uncertainty quantification would substantially strengthen the paper. The novelty relative to Time-MMD and MTBench should also be made crisper; the current comparison in Table 1 relies on self-reported numbers and the 'RP' column conflates protocol availability with actual protocol use."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the dataset suite itself is a real contribution: 869,466 series across eight domains with aligned text, image, and metadata, plus a pre-release-based cold-start protocol, goes well beyond Time-MMD and MTBench in scale and in forecasting-specific design. The paper is honest about mixed results, ships detailed per-dataset statistics, and the Kaggle release makes it usable. If I built a multimodal forecasting benchmark tomorrow, I would start from MoTime. Second, the paper's load-bearing empirical claim—that external modalities improve cold-start forecasting—has a likely leakage problem that needs fixing before the numbers can be trusted. For WikiPeople, the time series end in mid-2017 but the Wikipedia summaries were scraped around 2025, and those summaries routinely contain deaths, elections, and later career events. The cold-start prompt feeds that text to a GPT model that \"forecasts\" past page views. AmazonReview has the same issue in a subtler form: the product metadata includes aggregate ratings computed over the full 2013-2023 period, which post-dates the cold-start windows. Neither the retrieval baseline nor the average baseline has access to that future information, so the reported GPT-vs-average gap in Figure 2 may be partly leakage, not modality utility. This is addressable with a timestamp audit and filtering, but it has to be done. The varying-history experiments have a softer weakness: the differences between unimodal and multimodal variants are often at the third decimal, there are no error bars or significance tests, and the multimodal variants add parameters, so any gain could be capacity rather than modality. The paper's own frame is benchmark infrastructure, so I would not demand a new method, but the modality-utility claim in the abstract needs stronger support. Minor issues: a couple of references are wrong (Chronos is misattributed), and the short/long history thresholds are asserted rather than justified. None of this changes the core value of the suite. The paper is for anyone building or testing context-aware forecasting models, especially cold-start methods, and it deserves peer review. My recommendation: send it out, but require the leakage audit, a re-run or re-analysis of cold-start with timestamp-safe text, and either confidence intervals or a capacity-matched ablation before acceptance.","headline":"A genuinely useful multimodal forecasting suite, but the cold-start claim rests on scraped text that hasn't been checked for future leakage, and the empirical support for modality gains is thinner than the abstract suggests.","tokens_in":25591,"tokens_out":1817,"would_cite":true,"duration_ms":19511,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoTime benchmark: text and image context improve forecasts across 869,466 series, including cold-start cases.","keywords":["multimodal time series forecasting","dataset suite","cold-start forecasting","varying-history forecasting","entity-centric modalities","text-conditioned forecasting","image-conditioned forecasting","retrieval-augmented generation"],"falsifier":"Take the MovieLens and WikiPeople splits, crawl the same text sources again, and check whether any summary references events dated after the forecast window (for example, a Wikipedia summary mentioning a person's death year after the series ends). Re-run the cold-start and short-history experiments with those sentences redacted; if the multimodal advantage shrinks or disappears, the reported gains are at least partly leakage rather than genuine use of context.","tokens_in":24638,"feed_emoji":"📊","tokens_out":5584,"duration_ms":48151,"temperature":0.7,"pith_summary":"MoTime is a dataset suite that pairs 869,466 item-level time series across eight domains with static external modalities—text descriptions, metadata, and images—and defines two evaluation protocols: varying-history forecasting and cold-start forecasting where no history exists. The paper's central claim is that external modalities can improve forecasting performance in both scenarios, with the clearest gains on short or sparse series where the temporal signal alone is weak. This matters because most forecasting benchmarks are unimodal, so there has been no large-scale, reusable way to test whether contextual information such as an item's category, description, or image helps a model anticipate popularity dynamics. The suite is built by re-purposing recommender-system and web-traffic sources into item-centric series, aligned to modalities through consistent identifiers, and is released publicly so the field can evaluate context-aware forecasting systematically.","feed_headline":"Text and images lift forecasting on 869K-series benchmark","feed_subtitle":"MoTime's cold-start and short-history tests show external context matters most when history runs out.","key_machinery":"The load-bearing object is the MoTime suite itself: eight datasets transformed into item-centric daily or high-frequency popularity series with aligned static modalities, built by aggregating user-item interactions (PixelRec, TaobaoFashion, AmazonReview, Tianchi, MovieLens) or repurposing web and news traffic logs (News, WikiPeople). The two evaluation scenarios are the second piece of machinery: a varying-history protocol that splits training series into long and short groups to measure marginal modality utility, and a cold-start protocol that masks all but the first valid observation so forecasts must come from external signals alone. For cold-start, the mechanism is a retrieval-augmented generation pipeline: frozen LLM embeddings, cosine similarity over a retrieval base, top-k neighbor series and metadata composed into a structured prompt, and an LLM generating the forecast. The two proposed model variants, MultiPatchTST and MultiWPMixer, use a time series encoder plus a frozen LLM encoder to fuse modalities, with images converted to captions before encoding.","core_discovery":"The paper introduces MoTime as the largest publicly available multimodal time series forecasting suite of its kind, covering 869,466 series with text, image, or metadata alignment and reusable protocols. In varying-history forecasting, it adapts PatchTST and WPMixer into multimodal variants by concatenating a time series encoder's patch embeddings with frozen-LLM text (and image-caption) embeddings; in cold-start forecasting, it uses a retrieval-augmented pipeline where a target entity's text description retrieves the top-k most similar entities and an LLM generates the forecast from their series and metadata. The empirical finding is that external modalities generally improve performance in both scenarios, but the benefit is dataset- and horizon-dependent: gains are notable for short series on some datasets, and cold-start retrieval beats a simple average baseline on most of the six tested datasets. The authors frame the contribution as infrastructure: a reusable suite and scenario-driven evaluation rather than a new forecasting model.","pith_inferences":["I read the paper as evidence that a series' identity—what the object is—is a learnable prior that matters most exactly when the observed signal is thinnest; a testable extension would be ablating modality type per dataset to see which channel carries the cold-start signal.","The retrieval-augmented cold-start design is effectively nearest-neighbour forecasting in an embedding space; a direct implication is that retrieval quality, not generation, sets the ceiling, so a similarity-weighted average over retrieved series might close much of the gap to LLM generation on datasets where the two already match.","The authors acknowledge that short series were not upsampled and sparsity was not systematically varied; a natural follow-up experiment would control sparsity and history length independently to separate their contributions to modality utility.","Because text summaries for MovieLens and WikiPeople were crawled from current public sources without a timestamp audit, the reported cold-start gains could be inflated by post-forecast information; the suite should ship with crawl dates or content-revision dates so users can audit leakage."],"forward_implications":["If the central claim holds, cold-start forecasting—predicting an item's popularity before any observations exist—becomes a tractable, benchmarkable task rather than a niche setting.","Static entity-level text and images can partially substitute for missing history, which would let recommender-system logs be reused as forecasting benchmarks wherever metadata already exists.","Because modality gains vary with history length, horizon, and sparsity, future benchmarks should report short/long and horizon-specific scores rather than a single aggregate, or they will hide where context actually helps.","The 869,466-series scale gives multimodal forecasting a training and evaluation corpus large enough to support large-model experiments analogous to those in unimodal forecasting."],"supporting_citations":[{"why":"Supplies the PixelRec user-interaction data that MoTime converts into long, sparse daily view series with thumbnails and titles.","marker":"[10]"},{"why":"Supplies the Amazon Reviews 2023 release from which the 29-category AmazonReview forecasting sub-datasets are derived.","marker":"[19]"},{"why":"Supplies the News dataset's 20-minute-interval popularity series and its headline, topic, and sentiment metadata.","marker":"[34]"},{"why":"Supplies the Wikipedia web-traffic source from which the WikiPeople multichannel view-count series and external summaries are taken.","marker":"[53]"},{"why":"Provides Time-MMD, the prior multimodal benchmark that MoTime compares against in scale and reusable-protocol terms.","marker":"[28]"},{"why":"Provides PatchTST, the patch-embedding architecture that MultiPatchTST extends with multimodal fusion.","marker":"[37]"},{"why":"Provides the dual-tower time-series-plus-frozen-LLM architecture that inspires the proposed multimodal model variants.","marker":"[63]"},{"why":"Provides the cold-start retrieval pipeline that MoTime adapts for retrieval-augmented generation forecasting.","marker":"[61]"}],"fun_headline_variants":["MoTime: 869K multimodal series lift forecasting","Text and images rescue cold-start time series","Multimodal data helps most when history is scarce","MoTime benchmark: external context boosts forecasts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The externally scraped text for MovieLens and WikiPeople—movie overviews and Wikipedia summaries—is assumed to contain no information from after the forecast period, but the paper performs no timestamp or content-revision audit, so a summary mentioning later events (such as a person's death or a film's later reception) could hand the model future knowledge and inflate the reported cold-start and short-history gains.","fun_headline_variants_meta":{"raw":{"variants":["MoTime: 869K multimodal series lift forecasting","Text and images rescue cold-start time series","Multimodal data helps most when history is scarce","MoTime benchmark: external context boosts forecasts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1716,"prompt_tokens":860,"completion_tokens":856,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":806}},"tokens_in":476,"tokens_out":856,"duration_ms":6090,"temperature":1.0,"reasoning_tokens":806,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:24:23.283854+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the MovieLens and WikiPeople splits, crawl the same text sources again, and check whether any summary references events dated after the forecast window (for example, a Wikipedia summary mentioning a person's death year after the series ends). Re-run the cold-start and short-history experiments with those sentences redacted; if the multimodal advantage shrinks or disappears, the reported gains are at least partly leakage rather than genuine use of context.","supporting_citations":[{"cited_title":"An Image Dataset for Benchmarking Recommender Systems with Raw Pixels","cited_arxiv_id":"2309.06789","evidence_quote":"Supplies the PixelRec user-interaction data that MoTime converts into long, sparse daily view series with thumbnails and titles."},{"cited_title":"Multi-source social feedback of online news feeds, 2018","cited_arxiv_id":null,"evidence_quote":"Supplies the News dataset's 20-minute-interval popularity series and its headline, topic, and sentiment metadata."},{"cited_title":"Web traffic forecasting with cnn and fibonacci median","cited_arxiv_id":null,"evidence_quote":"Supplies the Wikipedia web-traffic source from which the WikiPeople multichannel view-count series and external summaries are taken."},{"cited_title":"Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, and B","cited_arxiv_id":null,"evidence_quote":"Provides Time-MMD, the prior multimodal benchmark that MoTime compares against in scale and reusable-protocol terms."},{"cited_title":"A time series is worth 64 words: Long-term forecasting with transformers","cited_arxiv_id":null,"evidence_quote":"Provides PatchTST, the patch-embedding architecture that MultiPatchTST extends with multimodal fusion."},{"cited_title":"Unveiling the potential of text in high-dimensional time series forecasting","cited_arxiv_id":null,"evidence_quote":"Provides the dual-tower time-series-plus-frozen-LLM architecture that inspires the proposed multimodal model variants."},{"cited_title":"Ccwtf: Codebase for cold-start and varying-length time series forecasting","cited_arxiv_id":null,"evidence_quote":"Provides the cold-start retrieval pipeline that MoTime adapts for retrieval-augmented generation forecasting."}],"review_version":1}