REVIEW 3 major objections 6 minor 1 cited by
FinMultiTime: A Four-Modal Bilingual Dataset for Financial Time-Series Analysis
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read FinMultiTime offers four aligned data modalities for stock forecasting across US and Chinese markets.
desk verdict New dataset artifact, but the main scale claim is contradicted by the paper's own Table 9, and the table preprocessing leaks future information. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the temporally aligned four-modal record: for each stock, daily OHLCV price series, semi-annual candlestick charts labeled with a 1–5 trend score by GPT-4.1, quarterly/annual structured financial tables, and daily news sentiment scores (1–5) from LSA-summarized articles. The alignment is what carries the argument: each modality is matched to the same trading calendar, with period-end financial figures forward-filled to daily prices, so that a model can consume all four streams as a joint input. The authors' experiments then vary which modalities are present and how many stocks are used, attributing the resulting error reductions to the dataset's scale and modality coverage.
What would settle it
Run the 70-stock experiments again with financial tables aligned only from their public filing dates (no forward-fill into earlier days), and compare MSE/MAE against the forward-filled version. If the fundamental-table rows of Tables 7 and 8 lose their advantage, the dataset's temporal alignment is contaminated by look-ahead bias.
Extended reading notes
Core claim
FinMultiTime is built to show that heterogeneous financial signals can be temporally aligned across two markets and languages, and that this alignment pays off in forecasting. The core discovery is empirical: incorporating news sentiment, fundamental tables, and chart-based trend labels alongside price series consistently lowers prediction error and improves trend-direction accuracy, while increasing the number of stocks in training further improves performance. The authors position the dataset as a benchmark and training resource that existing text-plus-price datasets cannot provide, because none combine all four modalities at this scale across US and Chinese markets.
Load-bearing premise
The preprocessing assumes that quarter-end financial figures can be attached to every trading day in that quarter, even though those figures were not public until later; if that forward-fill leaks future information, the table modality's reported gains are inflated.
Editorial extensions
If this is right
- Researchers can train and compare multimodal forecasting models on a shared bilingual benchmark spanning two major markets.
- Models that fuse sentiment, fundamentals, and chart trends with prices should beat price-only baselines at short horizons, as the paper's tables show.
- The dataset's scale (112.6 GB) makes it usable for pre-training or fine-tuning large financial language models, not just small recurrent or convolutional networks.
- The reproducible pipeline implies the dataset can be extended past 2025, allowing longitudinal studies of how predictive signals change across market regimes.
- The moderate gains from fusion in Transformer models suggest that fusion architecture, not just data, is the next bottleneck.
Reading between the lines
- One implication the paper leaves implicit is that the reported gains from the table modality rest on the forward-filling choice; a strict point-in-time version of the tables would likely show smaller gains, since quarter-end figures were not public on earlier trading days.
- A testable extension is to run the same 70-stock comparison with a lagged table alignment (using a figure only after the filing date) and check whether the fundamental-table row still beats time-series-only.
- The sentiment and trend scores are produced by GPT-4.1 and LSA, so a natural robustness check is to re-label a sample with a different model or human annotators and measure how much of the prediction gain survives.
- The trend-label distribution in Figure 4 shows strong market skew, which suggests the image modality's contribution may partly reflect regime-level drift rather than stock-specific signal.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces FinMultiTime, a four-modality dataset for financial time-series forecasting that aligns financial news, structured financial tables, K-line technical charts, and stock price time series across U.S. (S&P 500) and Chinese (HS 300) markets. The dataset comprises roughly 112.6 GB, covers 2009–2025, and is claimed to be the first large-scale cross-market multimodal financial time-series resource. The authors perform experiments on a 70-stock subset to support three claims: (1) scale and data quality markedly boost prediction accuracy; (2) multimodal fusion yields moderate gains in Transformer models; and (3) a fully reproducible pipeline enables seamless dataset updates. The manuscript provides extensive dataset statistics, baseline comparisons, and an appendix on construction challenges and ethics.
Significance. If the dataset were constructed and validated as described, FinMultiTime would fill a real gap: existing financial multimodal datasets are mostly text+price, single-market, and smaller; none combines all four modalities at this scale in two languages. The authors also provide a useful comparison table and release the data on HuggingFace. However, the paper's headline empirical claim is contradicted by its own Table 9, and the table preprocessing introduces look-ahead bias. As a result, the dataset's value as a benchmark is not established by the experiments, and the claimed scale benefit cannot be credited. The resource may still be useful to the community after correction, but the current manuscript does not support its main findings.
major comments (3)
- [§4.3, Table 9] The text in §4.3 and the Conclusion claim that model performance 'consistently improves' as the number of stocks grows from 5 to 35, but Table 9 shows the opposite for most entries: FTS-Text-MoE on S&P 500 has MAE increasing from 0.0894 (5 stocks) to 0.3299 (15) to 0.3914 (35) and MSE increasing from 0.2263 to 0.4818 to 0.5344; ChatTime on HS300 has MAE increasing from 0.2647 to 0.4804 to 0.5643; CALF on S&P 500 has MAE increasing from 0.3763 to 0.4996 to 0.5503. Only CALF on HS300 improves between 5 and 15 stocks. This direct contradiction invalidates the abstract's finding (1), the Introduction's third bullet, and the Conclusion's first finding. The scale claim needs to be withdrawn or the experiment rerun with a proper protocol (e.g., multiple seeds/splits, reported confidence intervals).
- [§2.2] The structured-table preprocessing forward-fills period-end financial figures to all trading days within the reporting window, matching quarter-end values to closing prices on the last trading day and then applying them to earlier days in the quarter. Because quarter-end financial statements are not public until after the period ends, this introduces look-ahead bias: models using the table modality on days before the filing date have access to future information. This contaminates the temporal alignment promised for FinMultiTime and inflates the Fundamental Table gains reported in Tables 7 and 8. The dataset should be rebuilt with point-in-time (lagged) fundamentals or explicit filing dates.
- [Table 6] Table 6 reports 299,923 'Semi-annual trend images' for each 35-stock market, which is 8,569 images per stock, yet images are generated from six-month windows (about 50 per stock over the 2009–2025 span, consistent with Table 2's ratios of ~46–65 images per stock). The same number 299,923 is also listed as 'Daily time-series points,' suggesting the image count is a copy-and-paste error. Since Table 6 is the basis of the experimental subset description, this inconsistency must be corrected.
minor comments (6)
- [Abstract vs. Introduction] The abstract says 'Covering 5,105 stocks' while the Introduction and Table 2 imply 5,586 (4,694 S&P 500 plus 892 HS300); the submission body's abstract also gives 5,586. The number should be reconciled.
- [§4.1.1] The selection of the '35 most influential constituents' is not defined; no objective criterion for 'influential' is provided, which limits reproducibility of the experimental subset.
- [Introduction and Appendix C] The reproducibility claim is undercut by the statement that 'The complete data collection, preprocessing pipeline, and example code will be available soon'; only the dataset link is provided, not the code.
- [Figures 2 and 3] The prompt text in Figures 2 and 3 appears duplicated; the same system/user prompt is shown verbatim under both figures, which is likely a layout error.
- [§4.2 and Figure 4] The image trend labels are produced by prompting GPT-4.1 on candlestick charts generated from the same price series used as the time-series modality; the paper should clarify that this modality is not an independent source of information but a non-linear summary of the price series.
- [Table 9] The table caption states 'Best scores per row are highlighted,' but no highlighting is visible in the submitted version; please add the highlighting or remove the statement.
Circularity Check
No circular derivation found: the dataset's modalities are constructed from external or historical data, not from the prediction target. The only self-citation is a non-load-bearing baseline, and the paper's internal inconsistencies are correctness concerns, not circularity.
full rationale
The paper's claimed 'predictions' are empirical evaluations of a dataset, not derivations whose outputs equal their inputs by construction. The image modality is explicitly generated from the same price series (Figure 1 caption: 'the images are generated from the six-month K-line charts of stock prices') and then labeled by GPT-4.1 into trend scores; this makes the image feature a nonlinear transform of historical price data, which is standard feature engineering rather than circularity, because the forecast target is future price, not the past trend label. News sentiment scores come from GPT-4.1 summaries of external news articles, and financial tables come from SEC and Tushare filings; neither is defined in terms of the prediction target. The scale experiment (Table 9) contains a serious internal inconsistency: the text claims 'model performance consistently improves as the number of stocks increases from 5 to 35,' but the table shows generally worsening MAE/MSE as the subset grows. This is a soundness and correctness problem, not a circularity of the kind where a claimed result reduces to its own input by definition. The only self-citation is the use of the authors' own FTS-Text-MoE [40] as one of three compared baselines; it is not load-bearing for the dataset's construction, for the multimodal fusion comparisons, or for the scale claim. Under the hard rules requiring a quoted equivalence between output and input, no circular step can be exhibited. Score 2 reflects one minor self-citation that is not load-bearing; the central claims still have independent empirical content, even though that content is partly contradicted by the paper's own Table 9.
Assumptions & free parameters
assumptions (3)
- domain assumption GPT-4.1 sentiment and trend labels correctly reflect market content
- domain assumption Temporal alignment of daily prices, minute-level news, quarterly tables, and semi-annual images does not leak future information
- ad hoc to paper The 70 selected stocks represent the dataset's value
Cite this review
Pith. "Pith review of FinMultiTime: A Four-Modal Bilingual Dataset for Financial Time-Series Analysis." pith.science (2026). https://pith.science/paper/54V2M4DY
@misc{pith2026250605019,
author = {Pith},
title = {Pith review of: FinMultiTime: A Four-Modal Bilingual Dataset for Financial Time-Series Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/54V2M4DY}},
note = {Machine review of arXiv:2506.05019}
}
read the original abstract
Pure time series forecasting tasks typically focus exclusively on numerical features; however, real-world financial decision-making demands the comparison and analysis of heterogeneous sources of information. Recent advances in deep learning and large scale language models (LLMs) have made significant strides in capturing sentiment and other qualitative signals, thereby enhancing the accuracy of financial time series predictions. Despite these advances, most existing datasets consist solely of price series and news text, are confined to a single market, and remain limited in scale. In this paper, we introduce FinMultiTime, the first large scale, multimodal financial time series dataset. FinMultiTime temporally aligns four distinct modalities financial news, structured financial tables, K-line technical charts, and stock price time series across both the S&P 500 and HS 300 universes. Covering 5,105 stocks from 2009 to 2025 in the United States and China, the dataset totals 112.6 GB and provides minute-level, daily, and quarterly resolutions, thus capturing short, medium, and long term market signals with high fidelity. Our experiments demonstrate that (1) scale and data quality markedly boost prediction accuracy; (2) multimodal fusion yields moderate gains in Transformer models; and (3) a fully reproducible pipeline enables seamless dataset updates.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Overcoming the Modality Gap in Context-Aided Forecasting
A semi-synthetic dataset of 7 million context-augmented time series windows with verifier-filtered contexts enables transfer to real-world context-aided forecasting and suggests data quality—not architecture—was the b...
Reference graph
Works this paper leans on
-
[1]
Dogu Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models.arXiv preprint arXiv:1908.10063(2019)
arXiv 2019
-
[2]
Adebiyi A Ariyo, Adewumi O Adewumi, and Charles K Ayo. 2014. Stock price prediction using the ARIMA model. In2014 UKSim-AMSS 16th international conference on computer modelling and simulation. IEEE, 106–112
work page 2014
-
[3]
Luc Bauwens, Sébastien Laurent, and Jeroen VK Rombouts. 2006. Multivariate GARCH models: a survey.Journal of applied econometrics21, 1 (2006), 79–109
work page 2006
-
[4]
Lei Chai, Hongfeng Xu, Zhiming Luo, and Shaozi Li. 2020. A multi-source heterogeneous data analytic method for future price fluctuation prediction.Neu- rocomputing418 (2020), 11–20
work page 2020
-
[5]
Clayton Leroy Chapman, Lars Hillebrand, Marc Robin Stenzel, Tobias Deußer, David Biesner, Christian Bauckhage, and Rafet Sifa. 2022. Towards generating financial reports from tabular data using transformers. InInternational Cross- Domain Conference for Machine Learning and Knowledge Extraction. Springer, 221–232
work page 2022
-
[6]
Jou-Fan Chen, Wei-Lun Chen, Chun-Ping Huang, Szu-Hao Huang, and An-Pin Chen. 2016. Financial time-series data analysis using deep convolutional neural networks. In2016 7th International conference on cloud computing and big data (CCBD). IEEE, 87–92
work page 2016
-
[7]
Zihan Chen, Lei Nico Zheng, Cheng Lu, Jialu Yuan, and Di Zhu. 2023. Chatgpt informed graph neural network for stock movement prediction.arXiv preprint arXiv:2306.03763(2023)
arXiv 2023
-
[8]
Junyan Cheng and Peter Chin. 2024. Sociodojo: Building lifelong analytical agents with real-world text and time series. InThe Twelfth International Conference on Learning Representations
work page 2024
Show all 46 references
-
[9]
Zihan Dong, Xinyu Fan, and Zhiyuan Peng. 2024. Fnspid: A comprehensive financial news dataset in time series. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4918–4927
2024
-
[10]
Kui Fu and Yanbin Zhang. 2024. Incorporating Multi-Source Market Sentiment and Price Data for Stock Price Prediction.Mathematics12, 10 (2024), 1572
2024
-
[11]
Udit Gupta. 2023. GPT-InvestAR: Enhancing stock investment strategies through annual report analysis with large language models.arXiv preprint arXiv:2309.03079(2023)
2023 arXiv
-
[12]
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. 2021. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems34 (2021), 10944–10956
2021
-
[13]
Bryan Kelly, Dacheng Xiu, et al. 2023. Financial machine learning.Foundations and Trends®in Finance13, 3-4 (2023), 205–363
2023
-
[14]
Kyoung-jae Kim. 2003. Financial time series forecasting using support vector machines.Neurocomputing55, 1-2 (2003), 307–319
2003
-
[15]
Kelvin JL Koa, Yunshan Ma, Ritchie Ng, and Tat-Seng Chua. 2024. Learning to generate explainable stock predictions using self-reflective large language models. InProceedings of the ACM Web Conference 2024. 4304–4315
2024
-
[16]
Yaxuan Kong, Yiyuan Yang, Yoontae Hwang, Wenjie Du, Stefan Zohren, Zhangyang Wang, Ming Jin, and Qingsong Wen. 2025. Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement.arXiv preprint arXiv:2503.01875(2025)
2025 arXiv
-
[17]
Ross Koval, Nicholas Andrews, and Xifeng Yan. 2024. Financial Forecasting from Textual and Tabular Time Series. InFindings of the Association for Computational Linguistics: EMNLP 2024. 8289–8300
2024
-
[18]
Bjoern Krollner, Bruce Vanstone, and Gavin Finnie. 2010. Financial time series forecasting with machine learning techniques: A survey. InEuropean Symposium on Artificial Neural Networks: Computational Intelligence and Machine Learning. 25–30
2010
-
[19]
Moreno La Quatra and Luca Cagliero. 2020. End-to-end training for financial report summarization. InProceedings of the 1st Joint Workshop on Financial Narrative Processing and MultiLing Financial Summarisation. 118–123
2020
-
[20]
Geon Lee, Wenchao Yu, Kijung Shin, Wei Cheng, and Haifeng Chen. 2025. Time- cap: Learning to contextualize, augment, and predict time series events with large language model agents. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 18082–18090
2025
-
[21]
Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Prab- hakar Kamarthi, Aditya Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, et al. 2024. Time-mmd: Multi-domain multimodal dataset for time series analysis.Advances in Neural Information Process...
2024
-
[22]
Peiyuan Liu, Hang Guo, Tao Dai, Naiqi Li, Jigang Bao, Xudong Ren, Yong Jiang, and Shu-Tao Xia. 2025. Calf: Aligning llms for time series forecasting via cross- modal fine-tuning. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 18915–18923
2025
-
[23]
Xiao-Yang Liu, Guoxuan Wang, Hongyang Yang, and Daochen Zha. [n. d.]. Fin- GPT: Democratizing Internet-scale Data for Financial Large Language Models. InNeurIPS 2023 Workshop on Instruction Tuning and Instruction Following
2023
-
[24]
Xiao-Yang Liu, Hongyang Yang, Qian Chen, Runjia Zhang, Liuqing Yang, Bowen Xiao, and Christina Dan Wang. 2020. FinRL: A deep reinforcement learning library for automated stock trading in quantitative finance.arXiv preprint arXiv:2011.09607(2020)
2020 arXiv
-
[25]
Alejandro Lopez-Lira and Yuehua Tang. 2023. Can chatgpt forecast stock price movements? return predictability and large language models.arXiv preprint arXiv:2304.07619(2023)
2023
-
[26]
Mantas Lukauskas, Vaida Pilinkien ˙e, Jurgita Bruneckien ˙e, Alina Stundžien ˙e, Andrius Grybauskas, and Tomas Ruzgas. 2022. Economic activity forecasting based on the sentiment analysis of news.Mathematics10, 19 (2022), 3461
2022
-
[27]
Malkiel and Eugene F
Burton G. Malkiel and Eugene F. Fama. 1970. EFFICIENT CAPITAL MARKETS: A REVIEW OF THEORY AND EMPIRICAL WORK.The Journal of Finance25, 2 (1970), 383–417. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1540- 6261.1970.tb00518.x doi:10.1111/j.1540-6261.1970.tb00518.x
1970
-
[28]
Eliza Mik. 2017. Smart contracts: terminology, technical limitations and real world complexity.Law, innovation and technology9, 2 (2017), 269–300
2017
-
[29]
Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. 2020. Financial time series forecasting with deep learning: A systematic literature review: 2005–2019.Applied soft computing90 (2020), 106181
2020
-
[30]
Guizhu Shen, Qingping Tan, Haoyu Zhang, Ping Zeng, and Jianjun Xu. 2018. Deep learning with gated recurrent unit networks for financial sequence predictions. Procedia computer science131 (2018), 895–903
2018
-
[31]
Wataru Souma, Irena Vodenska, and Hideaki Aoyama. 2019. Enhanced news sentiment analysis using deep learning methods.Journal of Computational Social Science2, 1 (2019), 33–46
2019
-
[32]
Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, and Jianxin Liao. 2025. Chattime: A unified multimodal time series foundation model bridging numerical and textual data. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 3...
2025
-
[33]
Xinlei Wang, Maike Feng, Jing Qiu, Jinjin Gu, and Junhua Zhao. 2024. From news to forecast: Integrating event analysis in llm-based time series forecasting with reflection.Advances in Neural Information Processing Systems37 (2024), 58118–58153
2024
-
[34]
2005.Applied linear regression
Sanford Weisberg. 2005.Applied linear regression. Vol. 528. John Wiley & Sons
2005
-
[35]
Andrew Robert Williams, Arjun Ashok, Étienne Marcotte, Valentina Zantedeschi, Jithendaraa Subramanian, Roland Riachi, James Requeima, Alexandre Lacoste, Irina Rish, Nicolas Chapados, et al . 2024. Context is key: A benchmark for forecasting with essential textual information.a...
2024 arXiv
-
[36]
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2022. Timesnet: Temporal 2d-variation modeling for general time series analysis.arXiv preprint arXiv:2210.02186(2022)
2022 arXiv
-
[37]
Huizhe Wu, Wei Zhang, Weiwei Shen, and Jun Wang. 2018. Hybrid deep se- quential modeling for social text-driven stock prediction. InProceedings of the 27th ACM international conference on information and knowledge management. 1627–1630
2018
-
[38]
Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al . 2024. Finben: A holistic financial benchmark for large language models.Advances in Neural Information Processing Systems37 (2024), 95716–95743
2024
-
[39]
Frank Z Xing, Erik Cambria, and Roy E Welsch. 2018. Natural language based financial forecasting: a survey.Artificial Intelligence Review50, 1 (2018), 49–73
2018
-
[40]
Wenyan Xu, Dawei Xiang, Rundong Wang, Yonghong Hu, Liang Zhang, Jiayu Chen, and Zhonghua Lu. 2025. Learning Explainable Stock Predictions with Tweets Using Mixture of Experts.arXiv preprint arXiv:2507.20535(2025)
2025 arXiv
-
[41]
Yumo Xu and Shay B Cohen. 2018. Stock movement prediction from tweets and historical prices. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1970–1979
2018
-
[42]
Hang Yang, Yubo Chen, Kang Liu, Yang Xiao, and Jun Zhao. 2018. Dcfee: A document-level chinese financial event extraction system based on automatically labeled training data. InProceedings of ACL 2018, System Demonstrations. 50–55
2018
-
[43]
Runze Yang, Longbing Cao, Xin You, Kun Fang, Jianxun Li, and Jie Yang. 2025. Fourier Basis Mapping: A Time-Frequency Learning Framework for Time Series Forecasting.arXiv preprint arXiv:2507.09445(2025)
2025 arXiv
-
[44]
Yi Yang, Yixuan Tang, and Kar Yan Tam. 2023. Investlm: A large language model for investment using financial domain instruction tuning.arXiv preprint arXiv:2309.13064(2023)
2023 arXiv
-
[45]
Wentao Zhang, Lingxuan Zhao, Haochong Xia, Shuo Sun, Jiaze Sun, Molei Qin, Xinyi Li, Yuqing Zhao, Yilei Zhao, Xinyu Cai, et al. 2024. A multimodal foun- dation agent for financial trading: Tool-augmented, diversified, and generalist. InProceedings of the 30th ACM SIGKDD Confer...
2024
-
[46]
sharp price rise
Yanzhao Zou and Dorien Herremans. 2023. PreBit—A multimodal model with Twitter FinBERT embeddings for extreme price movement prediction of Bitcoin. Expert Systems with Applications233 (2023), 120838. A Bilingual News Summarize Algorithm In reference to FNSPID [ 9], we introduc...
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.