Pith. sign in

REVIEW 4 major objections 6 minor 47 references

FinDeepIndicator is a new benchmark for end-to-end financial indicator construction. Its central result: the best-performing system achieves only about 40% final-answer accuracy, with data collection as the primary bottleneck.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 00:18 UTC pith:AYXWWDZ7

load-bearing objection Solid, useful benchmark with a genuinely process-level evaluation; the data collection bottleneck claim is plausible but rests on underdocumented ground-truth data sources. the 4 major comments →

arxiv 2608.00764 v1 pith:AYXWWDZ7 submitted 2026-08-01 cs.AI

FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction

Chaoqun Yang , Fengbin Zhu , Xinyu Lin , Long Bai , Xiaoluan Liu , Ke-Wei Huang , Roger Zimmermann , Tat-Seng Chua This is my paper
classification cs.AI
keywords Deep ResearchFinancial AnalysisProcess-level Evaluationfinancial indicatorsdata collection bottleneckLLM-as-judgebenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces FinDeepIndicator, a benchmark that asks AI systems to construct financial indicators from scratch: given a natural-language question, an agent must state the indicator formula, retrieve the raw financial data, compute intermediate values, and deliver the final number. The authors' central claim is that this is the first benchmark to evaluate the entire construction process stage by stage, and that current search-equipped LLMs and deep research agents cannot do it reliably. Across 3,350 question-answer pairs covering 234 indicators, U.S. and Chinese markets, and 800 listed companies over ten years, the best system reaches only about 40% final-answer accuracy even though formula accuracy exceeds 70% for most models. The paper's main diagnostic conclusion is that the bottleneck is data collection, not financial knowledge: when formulas and raw data are supplied, accuracy jumps above 80% for fundamental and macroeconomic indicators.

Core claim

FinDeepIndicator's central finding is that state-of-the-art systems understand what financial indicators mean but cannot reliably execute the full construction pipeline. The paper breaks construction into four stages—formula specification, data collection, indicator calculation, and answer generation—and measures each one. It reports that formula specification accuracy generally exceeds 70%, data collection accuracy drops by roughly 40 points, and final-answer accuracy tops out near 40%. Error analysis attributes 87.7% of failures to data-stage mistakes such as retrieving incorrect values, missing critical fields, choosing the wrong time window, and mishandling units. Macroeconomic indicator

What carries the argument

The load-bearing mechanism is the four-stage process-level evaluation protocol. Model responses are parsed by an LLM into four structured components—formula, raw data table, indicator calculation table, and final answer—then scored separately: formulas by an LLM-as-judge for semantic and mathematical equivalence, and tables and answers by tolerance-aware cell-level numerical matching. The benchmark adds 170 templates at three difficulty levels and 234 indicators organized into 21 sub-categories, so errors can be localized to a specific stage rather than hidden in a final number.

Load-bearing premise

The whole benchmark rests on the reference tables: if the underlying raw data and computed ground-truth answers are wrong, incomplete, or drawn from an unspecified data source, every stage-wise accuracy number and the bottleneck conclusion shift.

What would settle it

Take a random sample of about 200 of the 3,350 QA pairs, recompute the reference indicator values directly from primary sources (company filings, exchange price histories, and official macro releases) for the exact periods in the questions, and compare with the paper's ground truth. Any material mismatch rate would invalidate the measured accuracies; exact agreement would confirm the benchmark's reference data.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Improving data retrieval, extraction, and normalization should matter more for financial agents than adding financial knowledge.
  • Stage-level scores let developers target agent-specific weak stages, such as one model's data collection versus another's calculation.
  • Macroeconomic indicator construction needs better handling of heterogeneous sources, reporting frequencies, and geographic definitions.
  • Technical indicators require better support for rolling-window and sequential computation even when retrieval is perfect.
  • Current agents are not reliable enough for unattended end-to-end financial analysis; the data stage needs human oversight.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the data-collection bottleneck generalizes beyond this benchmark, agent builders should invest in structured financial data APIs and verification tools before pursuing larger models.
  • The process-level evaluation method could transfer to other numerical domains where a formula, a retrieval step, and a computation chain are separable, such as scientific or engineering calculation tasks.
  • Because the benchmark was generated from templates, model performance may be optimistically biased if agents overfit to template patterns; a human-written held-out set with the same indicators would test whether the 40% ceiling reflects genuine difficulty.
  • A two-stage agent that verifies retrieved data against multiple sources before computing could plausibly recover a large share of the 40-point data-collection drop; the benchmark is set up to measure which stage such an intervention helps.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces FinDeepIndicator, a benchmark for evaluating Deep Research agents on end-to-end financial indicator construction. The benchmark decomposes the task into four stages (formula specification, data collection, indicator calculation, answer generation), covers 234 indicators and 3,350 QA pairs drawn from U.S. and Chinese markets over 10 years and 800 listed companies, and evaluates search-equipped LLMs and a ReAct-style agent on those stages. The headline results are that the best system reaches only about 40% final-answer accuracy, that formula specification is relatively strong (above 70% for most models), and that the largest stage-wise drop occurs at data collection, which the paper identifies as the primary bottleneck. The paper includes human validation of the response-parsing and formula-judging procedures, an ablation in which formulas and raw data are provided, and a 300-case error analysis attributing most failures to data-related errors.

Significance. The process-level decomposition is a genuinely useful contribution: most financial QA benchmarks either provide evidence in context or evaluate only final answers, so a benchmark that scores formula, data, calculation, and answer stages separately can support fine-grained diagnosis of financial agents. If the benchmark data and metadata are released, FinDeepIndicator could become a valuable resource for the community. The internal evaluation is also careful in several respects: template-based generation is controlled, expert validation is reported for parsing and formula evaluation, the ablation study directly tests the role of data provision, and the error analysis is concrete. There is no obvious circular parameter fitting or self-referential derivation. However, the central numerical claims rest on reference data tables whose provenance, dates, and exact construction are not disclosed, and the bottleneck conclusion is stated using metrics that mix cell-level and instance-level accuracy. These issues need to be resolved before the benchmark numbers and conclusions are fully interpretable.

major comments (4)
  1. [Sections 2.1.1, 2.1.4, 6] The ground-truth raw-data tables are the linchpin of every stage-wise accuracy in Figure 4, the ablation in Figure 8, and the error distribution in Table 2, but the paper never identifies the data providers, the retrieval dates, the exact 10-year window for the 800 companies, or the handling of restatements, reporting periods, and revised macroeconomic series. Section 6 only says 'publicly available financial and macroeconomic data.' Because agents retrieve from the live web, a static reference table from an unnamed source may differ from what a live search returns for legitimate reasons (restated filings, fiscal-versus-calendar year conventions, revised GDP series). Please publish the full provenance: source names and versions, snapshot dates, extraction scripts, the complete reference tables, and the company/date lists. Without these, none of the stage accuracies can be independently v
  2. [Section 2.2.2 and Section 3.2] The 'data collection is the primary bottleneck' claim is supported partly by comparing the formula-specification accuracy with the data-collection accuracy, but these metrics have different granularities. Formula and answer accuracies are instance-level, while data collection accuracy is defined as correctly matched cells divided by total cells in the reference table. A response with one missing cell in a ten-cell table loses 10% on the cell-level metric even if the question would still be answered correctly; conversely, an instance-level formula error can propagate into the data stage. To make the bottleneck claim conclusive, report per-instance stage success rates (e.g., the fraction of questions whose data table fully matches, and the fraction of questions whose data stage is correct conditional on formula correctness) and stage-conditional final-answer rates. This is also needed to i
  3. [Section 2.2.2 and Table 1] The tolerance-aware matching is central to the data, calculation, and answer metrics, but the paper only says the tolerance is 'predefined' with examples such as 0.01, 0.005, or 0.001. The actual threshold per indicator or field, the rule for choosing it, and the sensitivity of the headline numbers to it are not provided. Additionally, the LLM-as-judge for formula equivalence is never named, and the parsing LLM (DeepSeek-V4-Flash) is itself one of the evaluated models. Please disclose the judge model and its version, report the size and stratification of the human sample used in Table 1, and provide either the exact tolerance mapping or a sensitivity analysis over tolerances. This matters because the gap between formula and data stages, and the ~40% final-answer figure, could shift with these choices.
  4. [Sections 2.1.3, 3.3, 3.4] The paper does not report the composition of the 3,350 QA pairs across markets, indicator sub-categories, and difficulty levels, although Figures 5-7 compare cross-market and sub-category performance. It also says the QA pairs are 'sampled' from a large candidate pool without specifying the sampling procedure, seed, exclusion criteria, or constraints used during instantiation. A benchmark datasheet should be included: counts by market, category, sub-category, difficulty, and temporal coverage, plus the exact generation code. Without this information, the representativeness of the benchmark and the strength of the cross-market and sub-category conclusions cannot be assessed.
minor comments (6)
  1. [Section 3.1] Typo: 'Deeepseek-V4-Flash' should be 'DeepSeek-V4-Flash'.
  2. [Section 5] The opening sentence is grammatically incomplete: 'Covering 234 indicators and 3,350 QA pairs across the U.S. and Chinese markets.' should be merged with the following sentence or rephrased.
  3. [Section 3.6] Please specify the exact input for the ablation: were the formula and raw data inserted into the context in a fixed format? Did the agent still have web access? The claim that this 'removes the need for data collection' is only as strong as the prompt protocol.
  4. [Section 2.2.1] The human validation of parsing used 100 model responses; report how these were sampled across models and stages, and give per-component acceptance rates with confidence intervals rather than only the aggregate.
  5. [Section 2.2.2] The Alt-test section reports winning rate and advantage probability but not the sample size or the exact threshold used to pass. Please report counts and statistical uncertainty for Table 1.
  6. [Figures 4-8] No error bars or repeated-run variance are reported. Since both sampling and live web results are stochastic, please provide seeds, number of runs, or variance estimates so that model differences can be separated from noise.

Circularity Check

0 steps flagged

No circular reasoning found: benchmark ground truth is constructed independently of evaluated models; stage-wise conclusions are empirical measurements.

full rationale

The paper's derivation chain is self-contained. The benchmark is built by (1) collecting 234 financial indicators with documented formulas, raw data requirements, and calculation programs; (2) designing templates with configurable variables and difficulty levels; and (3) instantiating these templates by random sampling to generate 3,350 QA pairs. Ground-truth answers are computed from the indicator metadata and reference data tables, not from any model under evaluation. No parameter is fitted to model outputs, and no prediction is derived from a quantity that itself depends on the prediction. The evaluation protocol compares parsed model responses against fixed reference formulas, reference data tables, and reference answers; the LLM-based parsing and formula judging are independently validated by human experts (Table 1 and the Alt-test), so the shared use of an LLM for parsing does not make the ground truth depend on the evaluated models. The stage-wise accuracy figures and the 'data collection is the primary bottleneck' conclusion are empirical measurements, not consequences of the benchmark definitions. The only substantive weakness is the underspecified provenance of the reference data (Section 6 merely says 'publicly available financial and macroeconomic data'), but that is a data-construction and reproducibility risk, not a circularity: the reference tables are not defined in terms of the models' outputs and the paper does not cite itself to justify the benchmark's correctness. Self-citations appear only in related-work context and are not load-bearing. Accordingly, no circular step can be exhibited with a specific reduction, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The central results rest on two domain assumptions: that the reference data tables are correct and complete, and that the four-stage decomposition cleanly attributes errors. A further ad-hoc assumption is that the unnamed LLM judge for formula equivalence behaves like a human on the full benchmark; the paper only validates it on 100 responses. No invented entities are proposed, and one hand-chosen evaluation parameter (numerical tolerance) affects all reported accuracies.

free parameters (1)
  • numerical tolerance thresholds = 0.01, 0.005, 0.001 (unit unspecified)
    Hand-chosen for cell and answer matching; reported stage accuracies and the 'unreliable' conclusion depend on these values, and no sensitivity analysis is shown.
axioms (3)
  • domain assumption Raw financial and macroeconomic data for all 234 indicators, 800 companies, and 10 years are available from public sources and are correctly normalized into the reference tables.
    Invoked throughout benchmark construction (Section 2.1.1 and Section 2.1.4); the paper does not name the specific data sources or retrieval dates.
  • domain assumption The four-stage decomposition (formula, data, calculation, answer) faithfully represents the indicator construction process and that errors can be cleanly attributed to one stage.
    The whole evaluation protocol (Section 2.2) rests on this decomposition; if errors compound across stages, per-stage accuracy may be misattributed.
  • ad hoc to paper The LLM-as-judge for formula semantic equivalence agrees with human judgment on the full benchmark.
    The paper uses an LLM to judge formulas (Appendix A.3); while validated with the Alt-test on a 100-response sample, the judge's reliability on the full 3,350 responses is assumed.

pith-pipeline@v1.3.0-alltime-deepseek · 14321 in / 13656 out tokens · 119868 ms · 2026-08-05T00:18:35.750351+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction." pith.science (2026). https://pith.science/paper/AYXWWDZ7

@misc{pith2026260800764,
  author       = {Pith},
  title        = {Pith review of: FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AYXWWDZ7}},
  note         = {Machine review of arXiv:2608.00764}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Financial indicators are essential tools for transforming raw financial data into interpretable measures for various downstream tasks, such as valuation, risk assessment, and economic analysis. However, existing financial benchmarks largely focus on answer-level accuracy and often assume that relevant data are already provided, leaving the assessment of the intermediate process of indicator construction underexplored. In this work, we propose FinDeepIndicator, the first benchmark dedicated to evaluating Deep Research (DR) agents in end-to-end financial indicator construction. Specifically, FinDeepIndicator evaluates DR agents across four stages in indicator construction: formula specification, data collection, indicator calculation, and answer generation, and covers fundamental, technical, and macroeconomic indicators organized into 21 fine-grained sub-categories. It contains 3,350 curated question-answer (QA) pairs derived from both U.S. and Chinese markets, 10 years of historical financial data, and 800 listed companies. Extensive experiments on search-equipped Large Language Models (LLMs) and DR agents show that, while LLMs generally perform well in formula specification, their accuracy drops substantially during data retrieval and numerical execution. DR agents consistently outperform search-equipped LLMs, yet remain unreliable in realistic financial analysis settings. These findings provide insights for developing more capable and trustworthy DR agents in finance.

Figures

Figures reproduced from arXiv: 2608.00764 by Chaoqun Yang, Fengbin Zhu, Ke-Wei Huang, Long Bai, Roger Zimmermann, Tat-Seng Chua, Xiaoluan Liu, Xinyu Lin.

Figure 2
Figure 2. Figure 2: An overview of FinDeepIndicator. verification, and process-level evaluation. Based on the proposed evaluation framework and collected indicators, we construct FinD￾eepIndicator, the first benchmark specifically designed for assess￾ing DR agents in end-to-end financial indicator construction. The benchmark spans the U.S. and Chinese markets and incorporates 10 years of historical data from 800 publicly list… view at source ↗
Figure 3
Figure 3. Figure 3: Construction pipeline of FinDeepIndicator. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison on agent setting between [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance comparison on agent setting under [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Fine-grained performance analysis across different financial indicator sub-categories. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Performance comparison on agent setting when [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 30 canonical work pages

  1. [1]

    Antoine Bigeard, Langston Nashold, Rayan Krishnan, and Shirley Wu. 2025. Finance agent benchmark: Benchmarking LLMs on real-world financial research tasks.arXiv preprint arXiv:2508.00828(2025)

    Pith/arXiv arXiv 2025

  2. [2]

    Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annota- tors with LLMs. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vienna, Austria, 16051–1608...

    doi:10.18653/v1/2025.acl-long.782 2025

  3. [3]

    Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024. FinTextQA: A dataset for long-form financial question answering. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 6025–6047

    work page 2024

  4. [4]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. 2021. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 3697–3711

    work page 2021

  5. [5]

    Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. ConvFinQA: Exploring the chain of numerical rea- soning in conversational finance question answering. InProceedings of the 2022 conference on empirical methods in natural language processing. 6279–6292

    work page 2022

  6. [6]

    Jaeyoung Choe, Jihoon Kim, and Woohwan Jung. 2025. Hierarchical retrieval with evidence curation for open-domain financial question answering on standardized documents. InFindings of the Association for Computational Linguistics: ACL 2025. 16663–16681

    work page 2025

  7. [7]

    Chanyeol Choi, Jihoon Kwon, Alejandro Lopez-Lira, Chaewoon Kim, Minjae Kim, Juneha Hwang, Jaeseon Ha, Hojun Choi, Suyeol Yun, Yongjin Kim, et al

    work page

  8. [8]

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2025. DeepResearch Bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763(2025)

    Pith/arXiv arXiv 2025

  9. [9]

    Aparna Elangovan, Ling Liu, Lei Xu, Sravan Babu Bodapati, and Dan Roth. 2024. Considers-the-human evaluation framework: Rethinking human evaluation for generative large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1137–1160

    work page 2024

  10. [10]

    Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, Zhiqiang Liu, Sizhe Wang, Jinyi Niu, Chuqi Wang, Yanhui Wang, et al . 2025. FinEval: A chinese financial domain knowledge evaluation benchmark for large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Hum...

    work page 2025

  11. [11]

    Janghoon Han, Heegyu Kim, Changho Lee, Dahm Lee, Min Hyung Park, Hosung Song, Stanley Jungkyu Choi, Moontae Lee, and Honglak Lee. 2025. DEER: A comprehensive and reliable benchmark for deep-research expert reports.arXiv e-prints(2025), arXiv–2512

    work page 2025

  12. [12]

    LIANG HU, Jianpeng Jiao, Jiashuo Liu, Dongyuan Mutu, Yanle Ren, Zhoufutu Wen, Kaiyuan Zhang, Xuanliang Zhang, Xiang Gao, Tianci He, FEI HU, Yali Liao, Zaiyuan Wang, Jingkai Liu, Sun Daibin, Ziqing Zeng, Zhiyuan Zeng, Chenghao Yang, Qianyu Yang, Mingren Yin, Ge Zhang, Xinyi zhang, Xiying ZHAO, Zhu Zhenwei, Hongseok Namkoong, and Wenhao Huang. 2026. FinSear...

    work page 2026

  13. [13]

    Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Huichi Zhou, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, et al . 2025. Deep research agents: A systematic examination and roadmap.arXiv preprint arXiv:2506.18096(2025)

    Pith/arXiv arXiv 2025

  14. [14]

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. FinanceBench: A new benchmark for financial question answering.arXiv preprint arXiv:2311.11944(2023)

    Pith/arXiv arXiv 2023

  15. [15]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-R1: Training LLMs to rea- son and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516(2025)

    Pith/arXiv arXiv 2025

  16. [16]

    Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2025. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technol...

    2025

  17. [17]

    Viet Lai, Michael Krumdick, Charles Lovering, Varshini Reddy, Craig Schmidt, and Chris Tanner. 2025. SEC-QA: A systematic evaluation corpus for financial QA. InProceedings of The 10th Workshop on Financial Technology and Natural Language Processing. 221–236

    work page 2025

  18. [18]

    Jinu Lee and Julia Hockenmaier. 2025. Evaluating Step-by-step Reasoning Traces: A Survey.arXiv preprint arXiv:2502.12289(2025)

    arXiv 2025

  19. [19]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. LLMs-as-judges: a comprehensive survey on LLM-based evaluation methods.arXiv preprint arXiv:2412.05579(2024)

    Pith/arXiv arXiv 2024

  20. [20]

    Minghao Li, Ying Zeng, Zhihao Cheng, Cong Ma, and Kai Jia. 2025. Report- Bench: Evaluating deep research agents via academic survey tasks.arXiv preprint arXiv:2508.15804(2025)

    Pith/arXiv arXiv 2025

  21. [21]

    Xiangyu Li, Xuan Yao, Guohao Qi, Fengbin Zhu, Kelvin JL Koa, Xiang Yao Ng, Ziyang Liu, Xingyu Ni, Chang Liu, Yonghui Yang, et al. 2026. FinDeepForecast: A Live Multi-Agent System for Benchmarking Deep Research Agents in Financial Forecasting.arXiv preprint arXiv:2601.05039(2026)

    arXiv 2026

  22. [22]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. InInternational Conference on Learning Representations, Vol. 2024. 39578–39601

    work page 2024

  23. [23]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment. InProceedings of the 2023 conference on empirical methods in natural language processing. 2511–2522

    work page 2023

  24. [24]

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: a benchmark for general AI assistants. InInternational Conference on Learning Representations, Vol. 2024. 9025–9049

    work page 2024

  25. [25]

    2007.The economics of money, banking, and financial markets

    Frederic S Mishkin. 2007.The economics of money, banking, and financial markets. Pearson education

    work page 2007

  26. [26]

    1999.Technical analysis of the financial markets: A comprehensive guide to trading methods and applications

    John J Murphy. 1999.Technical analysis of the financial markets: A comprehensive guide to trading methods and applications. Penguin

    work page 1999

  27. [27]

    2025.Introducing deep research

    OpenAI. 2025.Introducing deep research. https://openai.com/index/introducing- deep-research/

    work page 2025

  28. [28]

    2010.Financial statement analysis and security valuation

    Stephen H Penman and Stephen H Penman. 2010.Financial statement analysis and security valuation. McGraw-Hill/Irwin New York

    work page 2010

  29. [29]

    Varshini Reddy, Rik Koncel-Kedziorski, Viet Dac Lai, Michael Krumdick, Charles Lovering, and Chris Tanner. 2024. DocFinQA: A long-context financial reason- ing dataset. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 445–458

    work page 2024

  30. [30]

    Raj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. When FLUE meets FLANG: Benchmarks and large pretrained language model for financial domain. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2322–2335

    work page 2022

  31. [31]

    Yiqing Shen, Jingshu Zhang, Feng Chen, Kaiyuan Yan, and Hongguang Li. 2025. FinSearch: A Temporal-Aware Search Agent Framework for Real-Time Financial Information Retrieval with Large Language Models. InProceedings of the 6th ACM International Conference on AI in Finance. 10–17

    work page 2025

  32. [32]

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-searcher: Incentivizing the search capability in LLMs via reinforcement learning.arXiv preprint arXiv:2503.05592 (2025)

    Pith/arXiv arXiv 2025

  33. [33]

    2025.Deep Research is now available on Gemini 2.5 Pro Experi- mental.https://blog.google/products/gemini/deep-research-gemini-2-5-pro- experimental/

    Gemini Team. 2025.Deep Research is now available on Gemini 2.5 Pro Experi- mental.https://blog.google/products/gemini/deep-research-gemini-2-5-pro- experimental/

    work page 2025

  34. [34]

    2025.Deep research (Qwen-Deep-Research)

    Qwen Team. 2025.Deep research (Qwen-Deep-Research). https://www. alibabacloud.com/help/en/model-studio/qwen-deep-research

    work page 2025

  35. [35]

    Chris Van Der Lee, Albert Gatt, Emiel Van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. InProceedings of the 12th international conference on natural language generation. 355–368

    work page 2019

  36. [36]

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese

  37. [37]

    Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al . 2024. FinBen: A holistic financial benchmark for large language models.Advances in neural information processing systems37 (2024), 95716–95743

    work page 2024

  38. [38]

    arXiv preprint arXiv:2504.12516(2025)

    Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516(2025)

    Pith/arXiv arXiv 2025

  39. [39]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. ReAct: Synergizing Reasoning and Acting in Language Models. InThe eleventh international conference on learning representations

    work page 2022

  40. [40]

    Zhuohan Xie, Daniil Orel, Rushil Thareja, Dhruv Sahnan, Hachem Madmoun, Fan Zhang, Debopriyo Banerjee, Georgi Georgiev, Xueqing Peng, Lingfei Qian, et al. 2025. FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning.arXiv preprint arXiv:2506.02515(2025)

    Pith/arXiv arXiv 2025

  41. [41]

    Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. ProcessBench: Identifying process errors in mathematical reasoning. InProceedings of the 63rd Annual XXX, XX, XX, XX, XX Chaoqun Yang et al. Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1009–1024

    work page 2025

  42. [42]

    Wenlin Zhang, Xiaopeng Li, Yingyi Zhang, Pengyue Jia, Yichao Wang, Huifeng Guo, Yong Liu, and Xiangyu Zhao. 2025. Deep research: A survey of autonomous research agents.arXiv preprint arXiv:2508.12752(2025)

    Pith/arXiv arXiv 2025

  43. [43]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A question an- swering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th annual meeting of the Association for Computational Lin- guistics and the 11th international joint conference on natur...

    work page 2021

  44. [44]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. 2023. Judging LLM- as-a-Judge with MT-Bench and Chatbot Arena. InAdvances in Neu- ral Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Le...

    work page 2023

  45. [45]

    Jie Zhu, Junhui Li, Yalong Wen, and Lifan Guo. 2024. Benchmarking large lan- guage models on CFLUE-a Chinese financial language understanding evaluation dataset. InFindings of the Association for Computational Linguistics: ACL 2024. 5673–5693. A Appendix A.1 Indicator List See Table 3. A.2 Expert Review Criteria Human experts are instructed to evaluate wh...

    work page 2024

  46. [46]

    Fengbin Zhu, Xiang Yao Ng, Ziyang Liu, Chang Liu, Xianwei Zeng, Chao Wang, Tianhui Tan, Xuan Yao, Pengyang Shao, Min Xu, et al. 2025. FinDeepResearch: Evaluating deep research agents in rigorous financial analysis.arXiv preprint arXiv:2510.13936(2025)

    arXiv 2025

  47. [2025]

    InProceedings of the 6th ACM International Conference on AI in Finance

    FinAgentBench: A benchmark dataset for agentic retrieval in financial question answering. InProceedings of the 6th ACM International Conference on AI in Finance. 632–637

    work page