Predictability Analysis and Prediction of Discrete Weather and Financial Time-Series Data with a Hamiltonian-Based Filter-Projection Approach

The generalized Langevin equation (GLE), derived by projection from a general many-body Hamiltonian, exactly describes the dynamics of an arbitrary coarse-grained variable in a complex environment. However, analysis and prediction of real-world data with the GLE is hampered by slow transient or seasonal data components and time-discretization effects. Machine-learning (ML) techniques work but are computer-resource demanding and difficult to interpret. We show that by convolution filtering, time-series data decompose into fast, transient and seasonal components that each obey Hamiltonian dynamics and, thus, can be separately analyzed by projection techniques. We introduce methods to extract all GLE parameters from highly discretized time-series data and to forecast future data including the environmental stochasticity. For daily-resolved weather data, our analysis reveals non-Markovian memory that decays over a few days. Our prediction accuracy is comparable to commercial (weather.com) and ML long short-term memory (LSTM) methods at a reduced computational cost by a factor of $10^2-10^3$ compared to LSTM. For financial data, memory is very short-ranged and the dynamics effectively is Markovian, in agreement with the efficient-market hypothesis; consequently, models simpler than the GLE are sufficient. Our GLE framework is an efficient and interpretable method for the analysis and prediction of complex time-series data.