Pith. sign in

REVIEW 4 major objections 7 minor 46 references

Naive text fusion often hurts time-series forecasting; constraining the text signal through a low-rank bottleneck consistently fixes it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:40 UTC pith:ZZOBFJPX

load-bearing objection Broad, honest empirical study showing naive text fusion often hurts TS forecasting and low-rank constrained fusion helps; the missing error bars keep the headline gains from being statistically established. the 4 major comments →

arxiv 2603.22372 v3 pith:ZZOBFJPX submitted 2026-03-23 cs.LG cs.AI

Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion

Seunghan Lee , Jun Seo , Jaehoon Lee , Sungdong Yoo , Minjae Kim , Tae Yoon Lim , Dongwan Kang , Hwanil Choi
show 2 more authors
SoonYoung Lee Wonbin Ahn
This is my paper
classification cs.LG cs.AI
keywords multimodal time series forecastingconstrained fusionlow-rank adaptertext-guided forecastingnaive fusion failurerepresentation filteringplug-in fusion moduleLoRA-style bottleneck
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the standard way of adding text to a time-series forecasting model—simple addition or concatenation of text embeddings into the temporal representation—frequently makes forecasts worse than using no text at all. The authors attribute the degradation to uncontrolled integration of auxiliary information that may be irrelevant or contradictory to the series. They show that constrained fusion strategies, which control how much text enters the model, consistently beat naive fusion across over 20,000 experiments spanning nine datasets, fourteen time-series backbones, four text encoders, and four forecast horizons. Their proposed Controlled Fusion Adapter (CFA), a plug-in that compresses the text embedding through a low-rank bottleneck before adding it as a residual to the temporal representation, achieves the best average normalized error (0.256 vs 0.349 for the unimodal baseline) and the highest win-rates against the unimodal model. A sympathetic reader would take the claim as: the reason multimodal text fusion has seemed unreliable is not that text is unhelpful, but that it is being injected without a capacity constraint that forces it to align with temporal dynamics.

Core claim

The paper's central claim is that in multimodal time-series forecasting, the time series must remain the primary modality and any auxiliary text signal must be injected in a constrained manner; unconstrained injection is actively harmful. Concretely, the authors demonstrate that across a large experimental grid, naive additive and concatenative fusion at any layer often underperforms the unimodal baseline, sometimes diverging to errors more than ten times larger. In contrast, four constrained strategies—a learned gate, feature-wise linear modulation (FiLM), orthogonal projection, and their proposed CFA—consistently outperform naive fusion, and CFA additionally beats the unimodal baseline on

What carries the argument

The Controlled Fusion Adapter (CFA) is the central object: a LoRA-style low-rank bottleneck that compresses the text embedding into a low-dimensional subspace and then expands it back for residual addition to the time-series embedding. The mechanism is the rank restriction itself—since the down- and up-projections both have rank at most D/r, the composed text transformation lives in a subspace of dimension at most D/r, which bounds the capacity of the text signal to alter temporal representations. This low-rank constraint does the filtering work: in the paper's synthetic study it suppresses contradicting and irrelevant text (largest MSE improvement, +20.04%, occurs for irrelevant text) while

Load-bearing premise

The empirical superiority of constrained fusion over the unimodal baseline rests on a comparison protocol where each configuration is tuned over ten learning rates and only the best result is reported, without repeated-seed error bars; if that gap shrinks or reverses under fixed hyperparameters or across seeds, the 'consistent improvement' claim weakens.

What would settle it

A direct replication with fixed learning rates (or a small set) and multiple random seeds per configuration, reporting mean and variance of normalized MSE, would settle the claim: if CFA's advantage over the unimodal baseline falls within noise—especially on settings where naive fusion does not diverge—the paper's central conclusion would be materially weakened. A second falsifier: construct a dataset where text is deliberately highly informative (e.g., a single text token encodes the future trend) and show that a naive additive or concatenative fusion makes no trade-off; the claim predicts th

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners integrating text into time-series models should avoid simple additive or concatenative fusion, which the paper shows frequently underperforms the unimodal baseline.
  • A plug-in low-rank residual adapter can be added to any existing time-series backbone without retraining it from scratch, since only the adapter and a text projection are trained while the text encoder stays frozen.
  • Constrained fusion is robust to irrelevant text: when text is mismatched to the series, CFA degrades the least relative to the unimodal model.
  • The approach is efficient, adding under one percent parameter and FLOP overhead compared with the unimodal model, unlike some constrained alternatives such as gating that increase FLOPs by roughly 31%.
  • Because CFA generalizes across 14 backbones and 4 text encoders, the benefit is not tied to a specific architecture or language model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the low-rank bottleneck can be viewed as an inductive bias that forces the text pathway to compete for a limited number of 'directions' in the temporal representation space; the paper's filtering story is plausible but the evidence is correlational, so an alternative explanation—that the bottleneck merely regularizes the fusion layer—is not ruled out.
  • Editorial: if the rank restriction is truly the operative mechanism, then varying the bottleneck rank and measuring performance versus text informativeness across datasets would give a direct test; the sensitivity analysis in the paper shows stability across ranks r=2..32, which is consistent with the filtering story but does not isolate it.
  • Editorial: the same constrained-fusion principle should transfer to other auxiliary modalities such as vision or tabular context, and the paper flags this as future work; a testable prediction is that naive vision fusion will also underperform unimodal baselines on the same datasets.
  • Editorial: the reported averaged win-rates aggregate configurations that include divergence-level failures of naive fusion; a reader should check whether constrained fusion still beats unimodal on the subset of settings where naive fusion does not diverge.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper studies multimodal time-series forecasting with auxiliary text. It reports that naive fusion (additive or concatenative injection of text embeddings into a TS backbone) often degrades performance relative to a unimodal baseline, and that this can be remedied by 'constrained fusion.' The authors propose Controlled Fusion Adapter (CFA), a low-rank residual adapter that projects text through a bottleneck before adding it to TS representations. They evaluate 10 fusion strategies over an unusually broad grid (14 TS models × 4 text models × 9 datasets × 4 horizons), report averaged normalized MSE and win-rates, and include a synthetic toy experiment plus several analyses (representation similarity, temporal attribution, efficiency, effective rank) to support the proposed mechanism. The central claim is that CFA consistently outperforms both naive fusion methods and the unimodal baseline, and that constrained fusion in general is more robust than naive fusion.

Significance. If the central claim holds, the paper offers a practical and simple design rule: when injecting text into a time-series forecaster, use a low-rank residual bottleneck rather than direct addition or concatenation. The breadth of the empirical grid—over 2K settings and 10 fusion strategies—is a real strength, as is the release of code and the honest flagging of divergence cases in the full tables. The toy experiment with matching/contradicting/irrelevant text is a useful controlled check. However, the lack of repeated-seed variance and the loosely specified normalization make the headline 'consistent' claim statistically unquantified, and the 'theoretical perspective' in Appendix G is essentially a restatement of the low-rank construction rather than an explanation of why irrelevant text is filtered.

major comments (4)
  1. [Section 4; Table 4; Figure 3; Table 18] The evaluation protocol reports only the best result over 10 learning rates per setting, with no repeated seeds or variance statistics (Section 4: 'Each setting is evaluated over 10 learning rates, reporting the best result'). This is load-bearing because the headline 'consistent improvement' is an aggregate over many settings, and the paper's own per-horizon tables show small or reversed differences (e.g., Table 18, Nonstationary Transformer, Agriculture H=6: CFA 0.053 vs unimodal 0.051). Selection noise of this magnitude is larger than many reported gaps. Please provide repeated-seed results (mean±std over at least 3–5 seeds) for a representative subset or the full grid, and/or a paired bootstrap over settings, so the win-rates and average normalized MSE have confidence intervals. Without this, the 'consistent' claim is not statistically supported.
  2. [Appendix G; Section 3.3] The 'theoretical perspective' in Appendix G states that rank(W_up W_down) ≤ D/r, which is true by construction. The conclusion that 'the injected textual signal lies in a subspace of dimension at most D/r' and therefore 'suppresses irrelevant information' is an empirical claim, not a theorem. The toy experiment (Appendix E) is a demonstration, not a proof. As written, the theoretical argument is circular with the design of the adapter. Please either remove the 'theoretical perspective' label, or provide a formal analysis (e.g., with explicit assumptions about signal vs noise directions) that shows how the low-rank projection selectively attenuates irrelevant components.
  3. [Figure 3; Section 4.2] The aggregated comparisons use 'normalized MSE,' but the normalization procedure is never defined in the main text. This matters because dataset scales differ enormously (e.g., Security MSE ≈ 106, Agriculture ≈ 0.08 in Table 4). If 'normalization' is per-dataset division by some reference, the average can be dominated by a few high-scale datasets. Please specify the normalization formula explicitly, and report per-dataset and per-backbone win-rates or medians, so the 0.256 vs 0.349 overall gap can be interpreted correctly.
  4. [Section 5 [2]; Appendix E] The toy experiment used to support the low-rank filtering mechanism is a single synthetic dataset (N=1000) with a single TS backbone (Nonstationary Transformer), a single split, and no repeated-seed variance. The text-contribution ratio in Appendix E.2 is reported after an unexplained constant shift ((r−0.25745)×10^6), making the numbers difficult to interpret. To make the mechanistic claim convincing, the toy experiment should be run with at least a few seeds and ideally with one or two additional backbones, with error bars on both the MSE comparisons and the contribution ratios.
minor comments (7)
  1. [Figure 1 caption] The caption says 'Average over 2K settings,' but the experiment grid includes 9×14×4×4 = 2016 settings, which is indeed ~2K. Please clarify whether this is per fusion method or total.
  2. [Table 4] The footnote defining 'Div.*' appears at the bottom of the table but is not referenced in the caption or header. Consider adding a sentence in the caption explaining the divergence criterion.
  3. [Equation (2)] The cosine similarity formula in Section 5 [3] is mis-formatted: the inner product and norms are not typeset correctly. Also, the layer index L is overloaded with the lookback length L; use a different symbol for the layer count.
  4. [Section 4.2 [4]] In the text, 'Table 3d' should refer to Figure 3d. Also, Figure 5a in Section 5 [2] is labeled 'Table 5a' in the text; please correct the cross-references.
  5. [Appendix D, Table 9] The hyperparameter table uses inconsistent notation: d_model is listed as 512 for Transformer-based models, but the text describes DLinear as operating on C_in directly. Clarify the relationship between d_model and the channel dimension in the fusion equations.
  6. [Abstract and Introduction] The abstract claims 'over 20K experiments,' while Section 4 states '2K settings × 10 fusion methods.' This is consistent, but the wording '20K experiments' could be clearer—each setting is a hyperparameter configuration, not a repeated-seed trial.
  7. [Appendix I, Table 15] Typo: 'Algriculture' should be 'Agriculture' in both Table 15 and Table 16.

Circularity Check

0 steps flagged

No significant circularity: the central claims are held-out empirical benchmarks; Appendix G restates the low-rank adapter definition but is not load-bearing.

full rationale

The paper's central claims—that naive fusion often underperforms the unimodal baseline, that constrained fusion consistently beats naive fusion, and that CFA is best—are established by direct held-out test-set comparisons across many datasets, backbones, text encoders, and horizons. The baseline numbers (e.g., unimodal normalized MSE 0.349 vs CFA 0.256; win rates in Table 4) are measured, not derived from the definition of CFA. CFA's advantage is not a fitted parameter renamed as a prediction: its low-rank bottleneck is a fixed architectural choice, and its benefit is tested against actual unimodal and alternative fusion models. The toy experiment (Appendix E) is a controlled synthetic check comparing CFA with and without the bottleneck under matching, contradicting, and irrelevant text; the reported improvements are empirical outcomes, not consequences of a definition. Appendix G does contain a 'theoretical perspective' that essentially restates the mathematical fact rank(W_up W_down) ≤ D/r from the dimensions of the adapter and then interprets this as suppressing irrelevant information. That inference is more of a design rationale than a derivation, but it is not load-bearing: the empirical sections do not rely on Appendix G for their conclusions, and the paper's Limitations section explicitly concedes that 'our analysis remains largely empirical.' The dataset and tooling citations (e.g., Time-MMD) are external and not self-citations, and no uniqueness theorem or prior result by the same authors is invoked to force the choice of CFA. The best-of-10-learning-rate protocol without repeated-seed error bars is a legitimate statistical-rigor concern about selection noise, but it is a benchmarking methodology issue, not circularity: the predicted quantities are not defined in terms of the fitted hyperparameters. Overall, the derivation chain is self-contained with respect to circularity; any weakness lies in empirical robustness rather than in definitional equivalence.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The central claim rests on standard ML infrastructure (Time-MMD data and mmtslib implementations), a hand-chosen bottleneck rank r=8 (with r=4 often better), the best-of-10-learning-rate selection protocol, and the empirical assertion that the low-rank subspace filters text semantically. No exotic entities or physical constants are introduced. The free parameters are few but the experimental protocol assumptions carry most of the burden.

free parameters (2)
  • Bottleneck reduction ratio r (CFA) = r=8 (r=4 best in several cases, Appendix I)
    Controls the subspace dimension D/r of the text injection. The paper states r=8 is adopted 'considering both performance and computational efficiency', and sensitivity analysis shows r=4 often better. The central claim does not depend critically on r, but the reported CFA numbers use this hand-chosen value.
  • Fusion injection position (first/middle/last) = Per-setting best position aggregated; CFA applied at every encoder layer (middle)
    The comparison protocol tunes the injection position per setting for the baselines and reports the best of 10 learning rates. CFA uses a fixed residual application at each layer; the headline comparison therefore compares a fixed CFA design against tuned baselines—a modeling choice that affects the magnitude of the reported gap.
axioms (4)
  • domain assumption The Time-MMD multimodal datasets and their text pairing are a valid benchmark for multimodal TS forecasting (Section 4, Appendix B).
    All conclusions are aggregated over these 9 datasets; if the dataset/text pairing is unrepresentative of real multimodal forecasting, the generalization claim weakens. The paper relies on Time-MMD [21] for data, splits, and the mmtslib implementations.
  • domain assumption Selection of the best of 10 learning rates is a valid way to compare fusion methods (Section 4.1, Table 8).
    The protocol reports only the best configuration per setting; the unimodal baseline receives the same treatment, so the comparison is internally fair, but the absolute magnitudes and win-rates are conditional on this selection procedure. This is the load-bearing experimental assumption.
  • ad hoc to paper A low-rank projection suppresses irrelevant information while preserving useful information (Appendix G, Appendix E).
    The appendix proves only that the injected signal lies in a subspace of dimension D/r (rank inequality). That irrelevant text is filtered by this subspace, rather than merely attenuated uniformly, is an empirical claim tested only on the authors' synthetic toy dataset; the paper's Limitations section concedes the theoretical understanding is shallow.
  • domain assumption Frozen text encoders (BERT, GPT-2, Llama-3, Doc2Vec) provide usable text embeddings for the paired text (Section 4.1, Appendix A).
    The method never fine-tunes the text model and relies on the frozen embeddings being aligned with the TS dynamics; the paper shows robustness across the four encoders, so this assumption is partially validated empirically.
invented entities (1)
  • Controlled Fusion Adapter (CFA) independent evidence
    purpose: Injects textual information into TS representations via a low-rank residual connection to filter irrelevant text.
    CFA is a new module, but it makes no new physical/formal prediction outside the paper: its evidence is the empirical benchmark improvements on Time-MMD and the synthetic toy experiment, both within the paper. It is a design entity with in-paper falsifiable handles (benchmark performance), but no independent out-of-paper handle such as a predicted measurement in a different regime.

pith-pipeline@v1.3.0-alltime-deepseek · 96510 in / 8422 out tokens · 94421 ms · 2026-08-02T17:40:23.353568+00:00 · methodology

0 comments
read the original abstract

Recent advances in multimodal learning have motivated the integration of auxiliary modalities such as text or vision into time series (TS) forecasting. However, most existing methods provide limited gains, often improving performance only in specific datasets or relying on architecture-specific designs that limit generalization. In this paper, we show that multimodal models with naive fusion strategies (e.g., simple addition or concatenation) often underperform unimodal TS models, which we attribute to the uncontrolled integration of auxiliary modalities which may introduce irrelevant information. Motivated by this observation, we explore various constrained fusion methods designed to control such integration and find that they consistently outperform naive fusion methods. Furthermore, we propose Controlled Fusion Adapter (CFA), a simple plug-in method that enables controlled cross-modal interactions without modifying the TS backbone, integrating only relevant textual information aligned with TS dynamics. CFA employs low rank adapters to filter irrelevant textual information before fusing it into temporal representations. We conduct over 20K experiments across various datasets and TS/text models, demonstrating the effectiveness of the constrained fusion methods. Code is available at: https://github.com/seunghan96/cfa.

Figures

Figures reproduced from arXiv: 2603.22372 by Dongwan Kang, Hwanil Choi, Jaehoon Lee, Jun Seo, Minjae Kim, Seunghan Lee, Soonyoung Lee, Sungdong Yoo, Tae Yoon Lim, Wonbin Ahn.

Figure 1
Figure 1. Figure 1: Constrained vs. naive fusion. In this paper, we conduct extensive experiments across diverse datasets and models ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of multimodal fusion strategies for TS. (Left) Naive fusion applies simple additive or concatenation operators at first, middle, or last stages without considering modality relevance. (Right) Constrained fusion incorporates textual information in a controlled manner by considering its relevance to TS. CFA injects textual signals via a residual connection constrained to a low-rank subspace to fil… view at source ↗
Figure 3
Figure 3. Figure 3: Performance across diverse settings. (a), (b), and (c) show the performance (normalized MSE) by Dataset, TS model, and Text model, respectively. (d) shows the overall average perfor￾mance across all settings. CFA (⋆) consistently achieves the lowest MSE among fusion strategies. 4.2 Performance with Various Settings In this section, we evaluate the proposed method across diverse 1) datasets, 2) TS backbones… view at source ↗
Figure 5
Figure 5. Figure 5: Toy experiment on low-rank bottleneck. (a) CFA with a bottleneck consistently out￾performs the version without a bottleneck across all text types, with the largest gain observed for irrelevant text. (b) The text-contribution ratio at the adapter output shows that matching text is injected more strongly than contradicting text, indicating that it selectively suppresses conflicting signals. 5 Analysis In thi… view at source ↗
Figure 4
Figure 4. Figure 4: Irrelevant text experiments. A robust fusion strategy is expected to ignore unrelated information and rely primarily on TS representations, such that its performance does not substantially degrade compared to the unimodal setting. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Cosine similarity between representations of TS-only and TS+Text models. We apply multiple fusion strategies across diverse settings, where each dot represents a single fusion method. The best-performing method (red) does not consistently induce the largest representation shift from the unimodal baseline, highlighting the importance of controlled integration of textual information into temporal representat… view at source ↗
Figure 7
Figure 7. Figure 7: [4] Visualization of multimodal TS forecasting. Although the unimodal model captures the initial pattern, it fails to model the subsequent upward trend, whereas CFA accurately captures the rise in the later horizon In addition, some fusion strategies even fail to converge (e.g., concat-first), indicating that naive fusion can hinder effective learning. Figure 5b reports the text-contribution ratio (See App… view at source ↗
Figure 9
Figure 9. Figure 9: Rank correlation of MAE and effective rank. (a) Distribution of rank correlations across various settings, showing a positive relationship between MAE and effective rank. (b) Methods with higher effective rank generally achieve lower MAE, resulting in a positive correlation (ρ = 0.6727). t0 t1 t2 t3 t4 t5 t6 t7 Input Horizon Additive-first Additive-middle Additive-last Concat-first Concat-middle Concat-las… view at source ↗
Figure 8
Figure 8. Figure 8: Various fusion methods. [5] Temporal attribution analysis ( [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Illustration of the toy dataset. Each time step of the TS is paired with a text description, categorized as 1) matching, 2) contradicting, or 3) irrelevant. Matching descriptions align with the TS trend, contradicting ones oppose it, and irrelevant ones provide no useful signal. B. Model and horizon. We use the Nonstationary Transformer [22] as the TS backbone with an input length of L = 8 and a predictio… view at source ↗
Figure 11
Figure 11. Figure 11: Text-contribution ratio [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Forecast comparison for a matched (left) and a contradicting (right) text sample. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 14 linked inside Pith

  1. [1]

    Multivariate time series dataset for space weather data analytics.Scientific data, 7(1):227, 2020

    Rafal A Angryk, Petrus C Martens, Berkay Aydin, Dustin Kempton, Sushant S Mahajan, Sunitha Basodi, Azim Ahmadzadeh, Xumin Cai, Soukaina Filali Boubrahimi, Shah Muhammad Hamdi, et al. Multivariate time series dataset for space weather data analytics.Scientific data, 7(1):227, 2020

    2020

  2. [2]

    Context matters: Leveraging contextual features for time series forecasting

    Sameep Chattopadhyay, Pulkit Paliwal, Sai Shankar Narasimhan, Shubhankar Agarwal, and Sandeep P Chinchali. Context matters: Leveraging contextual features for time series forecasting. arXiv preprint arXiv:2410.12672, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Tsmixer: An all-mlp architecture for time series forecasting.TMLR, 2023

    Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik, and Tomas Pfister. Tsmixer: An all-mlp architecture for time series forecasting.TMLR, 2023

    2023

  4. [4]

    T3time: Tri-modal time series forecasting via adaptive multi-head alignment and residual fusion.arXiv preprint arXiv:2508.04251, 2025

    Abdul Monaf Chowdhury, Rabeya Akter, and Safaeid Hossain Arib. T3time: Tri-modal time series forecasting via adaptive multi-head alignment and residual fusion.arXiv preprint arXiv:2508.04251, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Towards spatio- temporal aware traffic time series forecasting

    Razvan-Gabriel Cirstea, Bin Yang, Chenjuan Guo, Tung Kieu, and Shirui Pan. Towards spatio- temporal aware traffic time series forecasting. In2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 2900–2913. IEEE, 2022

    2022

  6. [6]

    A. Das, W. Kong, A. B. Leach, S. Mathur, R. Sen, and R. Yu. Long-term forecasting with tide: Time-series dense encoder.arXiv Preprint arXiv:2304.08424, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InNAACL, 2018

    2018

  8. [8]

    Attention based multi-modal new product sales time-series forecasting

    Vijay Ekambaram, Kushagra Manglik, Sumanta Mukherjee, Surya Shravan Kumar Sajja, Satyam Dwivedi, and Vikas Raykar. Attention based multi-modal new product sales time-series forecasting. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3110–3118, 2020

    2020

  9. [9]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  10. [10]

    Gpt4mts: Prompt-based large language model for multimodal time-series forecasting

    Furong Jia, Kevin Wang, Yixiang Zheng, Defu Cao, and Yan Liu. Gpt4mts: Prompt-based large language model for multimodal time-series forecasting. InAAAI, 2024

    2024

  11. [11]

    Multi-modal time series analysis: A tutorial and survey

    Yushan Jiang, Kanghui Ning, Zijie Pan, Xuyang Shen, Jingchao Ni, Wenchao Yu, Anderson Schneider, Haifeng Chen, Yuriy Nevmyvaka, and Dongjin Song. Multi-modal time series analysis: A tutorial and survey. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V . 2, pages 6043–6053, 2025

    2025

  12. [12]

    Time-llm: Time series forecasting by reprogramming large language models

    Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. InICLR, 2024

    2024

  13. [13]

    Multi-modal forecaster: Jointly predicting time series and textual data

    Kai Kim, Howard Tsai, Rajat Sen, Abhimanyu Das, Zihao Zhou, Abhishek Tanpure, Mathew Luo, and Rose Yu. Multi-modal forecaster: Jointly predicting time series and textual data. arXiv:2411.06735, 2024. Preprint

    Pith/arXiv arXiv 2024

  14. [14]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  15. [15]

    Reformer: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020

    Pith/arXiv arXiv 2001

  16. [16]

    Le and Tomas Mikolov

    Quoc V . Le and Tomas Mikolov. Distributed representations of sentences and documents. In ICML, 2014. 10

    2014

  17. [17]

    Language in the flow of time: Time-series-paired texts weaved into a unified temporal narrative.arXiv preprint arXiv:2502.08942, 2025

    Zihao Li, Xiao Lin, Zhining Liu, Jiaru Zou, Ziwei Wu, Lecheng Zheng, Dongqi Fu, Yada Zhu, Hendrik Hamann, Hanghang Tong, et al. Language in the flow of time: Time-series-paired texts weaved into a unified temporal narrative.arXiv preprint arXiv:2502.08942, 2025

    arXiv 2025

  18. [18]

    Timi: Empower time series transformers with multimodal mixture of experts.arXiv preprint arXiv:2602.21693, 2026

    Jiafeng Lin, Yuxuan Wang, Huakun Luo, Zhongyi Pei, and Jianmin Wang. Timi: Empower time series transformers with multimodal mixture of experts.arXiv preprint arXiv:2602.21693, 2026

    arXiv 2026

  19. [19]

    Pa-rnet: Perturbation-aware reasoning network for multimodal time series forecasting.arXiv preprint arXiv:2508.04750, 2025

    Chanjuan Liu, Shengzhi Wang, and Enqiang Zhu. Pa-rnet: Perturbation-aware reasoning network for multimodal time series forecasting.arXiv preprint arXiv:2508.04750, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Timecma: Towards llm-empowered time series forecasting via cross-modality alignment.AAAI, pages arXiv–2406, 2025

    Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang, Lingzheng Zhang, Cheng Long, Ziyue Li, and Rui Zhao. Timecma: Towards llm-empowered time series forecasting via cross-modality alignment.AAAI, pages arXiv–2406, 2025

    2025

  21. [21]

    Time-mmd: Multi-domain multimodal dataset for time series analysis.Advances in Neural Information Processing Systems, 37:77888–77933, 2024

    Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Prabhakar Kamarthi, Aditya Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, et al. Time-mmd: Multi-domain multimodal dataset for time series analysis.Advances in Neural Information Processing Systems, 37:77888–77933, 2024

    2024

  22. [22]

    Liu et al

    Y . Liu et al. Non-stationary transformers for time series forecasting. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  23. [23]

    itransformer: Inverted transformers are effective for time series forecasting

    Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. InICLR, 2024

    2024

  24. [24]

    Koopa: Learning non-stationary time series dynamics with koopman predictors.Advances in neural information processing systems, 36:12271–12290, 2023

    Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series dynamics with koopman predictors.Advances in neural information processing systems, 36:12271–12290, 2023

    2023

  25. [25]

    Spectral text fusion: A frequency-aware approach to multimodal time-series forecasting

    Huu Hiep Nguyen, Minh Hoang Nguyen, Dung Nguyen, and Hung Le. Spectral text fusion: A frequency-aware approach to multimodal time-series forecasting. arXiv:2602.01588, 2026. Preprint

    Pith/arXiv arXiv 2026

  26. [26]

    Y . Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam. Patchtst: A patch-based time series transformer with channel independence.arXiv Preprint arXiv:2211.14730, 2023

    Pith/arXiv arXiv 2023

  27. [27]

    Unicast: A unified multimodal prompting framework for time series forecasting

    Sehyuk Park, Soyeon Caren Han, and Eduard Hovy. Unicast: A unified multimodal prompting framework for time series forecasting. arXiv:2508.11954, 2025. Preprint

    arXiv 2025

  28. [28]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. InAAAI, 2018

    2018

  29. [29]

    Language models are unsupervised multitask learners.OpenAI, 2019

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.OpenAI, 2019. Technical report

    2019

  30. [30]

    Stock price prediction using deep learning and frequency decomposition.Expert Systems with Applications, 169:114332, 2021

    Hadi Rezaei, Hamidreza Faaljou, and Gholamreza Mansourfar. Stock price prediction using deep learning and frequency decomposition.Expert Systems with Applications, 169:114332, 2021

    2021

  31. [31]

    Multimodal conditioned diffusive time series forecasting

    Chen Su, Yuanhe Tian, and Yan Song. Multimodal conditioned diffusive time series forecasting. arXiv preprint arXiv:2504.19669, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, 2017

    2017

  34. [34]

    Chattime: A unified multimodal time series foundation model bridging numerical and textual data

    Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, and Jianxin Liao. Chattime: A unified multimodal time series foundation model bridging numerical and textual data. InAAAI, 2025. 11

    2025

  35. [35]

    Timexer: Empowering transformers for time series forecasting with exogenous variables.Advances in Neural Information Processing Systems, 37:469–498, 2024

    Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yunzhong Qiu, Jianmin Wang, and Mingsheng Long. Timexer: Empowering transformers for time series forecasting with exogenous variables.Advances in Neural Information Processing Systems, 37:469–498, 2024

    2024

  36. [36]

    Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting

    Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. InNeurIPS, 2021

    2021

  37. [37]

    Promptcast: A new prompt-based learning paradigm for time series forecasting.IEEE Transactions on Knowledge and Data Engineering, 36(11):6851–6864, 2023

    Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting.IEEE Transactions on Knowledge and Data Engineering, 36(11):6851–6864, 2023

    2023

  38. [38]

    Context-aware probabilistic modeling with llm for multimodal time series forecasting

    Yueyang Yao, Jiajun Li, Xingyuan Dai, MengMeng Zhang, Xiaoyan Gong, Fei-Yue Wang, and Yisheng Lv. Context-aware probabilistic modeling with llm for multimodal time series forecasting. arXiv:2505.10774, 2025. Preprint

    Pith/arXiv arXiv 2025

  39. [39]

    Frequency-domain mlps are more effective learners in time series forecasting.Advances in Neural Information Processing Systems, 36:76656–76679, 2023

    Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequency-domain mlps are more effective learners in time series forecasting.Advances in Neural Information Processing Systems, 36:76656–76679, 2023

    2023

  40. [40]

    A. Zeng, S. Chen, L. Zhang, and Q. Xu. DLinear: Efficient linear models for time series forecasting.arXiv preprint arXiv:2205.13504, 2022

    Pith/arXiv arXiv 2022

  41. [41]

    Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting

    Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. InICLR, 2023

    2023

  42. [42]

    Time- vlm: Exploring multimodal vision-language models for augmented time series forecasting

    Siru Zhong, Weilin Ruan, Ming Jin, Huan Li, Qingsong Wen, and Yuxuan Liang. Time- vlm: Exploring multimodal vision-language models for augmented time series forecasting. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 ofPMLR, pages 78478–78497, 2025

    2025

  43. [43]

    Informer: Beyond efficient transformer for long sequence time-series forecasting

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021

    2021

  44. [44]

    Balm-tsf: Balanced multimodal alignment for llm-based time series forecasting

    Shiqiao Zhou, Holger Schöner, Huanbo Lyu, Edouard Fouché, and Shuo Wang. Balm-tsf: Balanced multimodal alignment for llm-based time series forecasting. arXiv:2509.00622, 2025. Preprint

    Pith/arXiv arXiv 2025

  45. [45]

    T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin. Film: Frequency improved legendre mem- ory model for long-term time series forecasting.Advances in Neural Information Processing Systems, 2022

    2022

  46. [46]

    The value is rising steadily

    Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. InICML, 2022. 12 A Details of Models We use the mmtslib package provided by Time-MMD [ 21]. We adopt 14 TS models and 4 text models implemented in the library.TS Models.We use 14 TS models from three...

    arXiv 2022