A hand-constructed O(ln L + T)-layer Transformer is shown to approximate low-rank hidden Markov models in-context, with lower layers extracting local history features and upper layers performing regression-based decoding.
Transformers and Their Roles as Time Series Foundation Models
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We give a comprehensive analysis of transformers as time series foundation models, focusing on their approximation and generalization capabilities. First, we demonstrate that there exist transformers that fit an autoregressive model on input univariate time series via gradient descent. We then analyze MOIRAI, a multivariate time series foundation model capable of handling an arbitrary number of covariates. We prove that it is capable of automatically fitting autoregressive models with an arbitrary number of covariates, offering insights into its design and empirical success. For generalization, we establish bounds for pretraining when the data satisfies Dobrushin's condition. Experiments support our theoretical findings, highlighting the efficacy of transformers as time series foundation models.
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Transformers as Multi-task Learners: Decoupling Features in Hidden Markov Models
A hand-constructed O(ln L + T)-layer Transformer is shown to approximate low-rank hidden Markov models in-context, with lower layers extracting local history features and upper layers performing regression-based decoding.