REVIEW 30 cited by
Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks
read the original abstract
Artificial Intelligence (AI) technology has emerged as a transformative force in financial analysis and the finance industry, though significant questions remain about the full capabilities of Large Language Model (LLM) agents in this domain. We present the Finance Agent Benchmark, featuring challenging and diverse real-world finance research problems that require LLMs to perform complex analysis using recent SEC filings. We construct the benchmark using a taxonomy of nine financial task categories, developed in consultation with experts from banks, hedge funds, and private equity firms. The dataset includes 537 expert-authored questions covering tasks from information retrieval to complex financial modeling, each validated through a rigorous review process to ensure accuracy and relevance. Moreover, we implement an agentic harness that equips LLMs with tools sufficient to produce accurate responses, including Google Search and EDGAR database access. Overall, the Finance Agent Benchmark provides a comprehensive testbed for measuring the progress of LLM-driven finance agents. Our evaluation reveals significant limitations in current AI capabilities - even the best-performing model (OpenAI o3) achieved only 46.8% accuracy at an average cost of $3.79 per query. This underscores the need for further advancements before reliable deployment in high-stakes finance settings.
Forward citations
Cited by 30 Pith papers
-
Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
CL-Bench is the first expert-validated benchmark for continual learning in frontier LLMs across six real-world domains, showing limited gains and that naive in-context learning outperforms dedicated memory systems.
-
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
AgentBeats implements agentified evaluation of diverse AI agents through standardized interfaces, validated at scale in a five-month competition with 298 judges and 467 subjects plus a coding case study.
-
Flaws in the LLM Automation Narrative
A new code-writing data analysis benchmark shows human experts outperforming a frontier LLM on average with lower performance variance.
-
BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents
BigFinanceBench is a workflow-grounded benchmark of 928 financial research tasks with point-weighted rubrics, where the best of ten tested agents scores 58.8% on derivation quality.
-
Herculean: An Agentic Benchmark for Financial Intelligence
Herculean benchmark shows frontier agents handle trading and market insights better than hedging and auditing workflows that demand state consistency and structured verification.
-
TacoMAS: Test-Time Co-Evolution of Topology and Capability in LLM-based Multi-Agent Systems
TacoMAS performs test-time co-evolution of agent capabilities and communication topology in LLM multi-agent systems via fast capability updates and slow meta-LLM topology edits, delivering 13.3% average gains over str...
-
BizCompass: Benchmarking the Reasoning Capabilities of LLMs in Business Knowledge and Applications
BizCompass is a dual-axis benchmark evaluating LLMs on business knowledge in finance, economics, statistics, and operations management, linked to analyst, trader, and consultant roles, with public datasets released af...
-
AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?
Self-evolving LLM agents improve under streaming evaluation only when the base model is strong enough; gains are largest in single-domain streams, and no method dominates across models.
-
AgentRadio: Passive Awareness for Long-Horizon Multi-Agent Collaboration
Background mention-waiting lets four Claude Code agents share discoveries mid-execution and reach 62.1% on SWE-Atlas QnA versus 32.3% for one agent.
-
Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
A unified corpus of 957k trial outcomes shows frontier progress is uneven and strict all-pass aggregation obscures capability and can reorder agents.
-
Frontier Financial Judgement: Can agents tell what might move a stock?
No evaluated AI agent can fully match professional analysts' newness/importance/direction labels on the new 82-case Frontier Financial Judgement benchmark; GPT-5.5 tops out at 52.4%.
-
Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
A new benchmark shows LLMs' financial calculation accuracy collapses without explicit formulas and degrades further when they must generate multi-metric tables.
-
FORCE-Bench: A Benchmark, Dataset, and Evaluation Harness for Agentic AI in Enterprise Finance
FORCE-Bench, a 251-question enterprise-finance benchmark with rubric scoring, shows a purpose-built Microsoft finance agent outperforming general-purpose agents on ERP and public-data tasks.
-
How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?
An empirical study evaluating tool-augmented LLM agents on 243 real-world energy analytics problems across data retrieval, knowledge interpretation, and quantitative modeling using domain-specific tools and multi-dime...
-
IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO
IPO Finance Agent creates a new LLM benchmark for IPO due diligence on SpaceX S-1 with contextual retrieval and automated rubrics, where Qwen 3.7 Max scores 79.4% at $0.30/query, outperforming Finance Agent v2 on accu...
-
IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO
IPO Finance Agent benchmarks LLMs on SpaceX S-1 questions with contextual retrieval and auto-generated rubrics, reporting up to 79.8% accuracy and better cost-efficiency than prior Finance Agent v2 entries.
-
Economy of Minds: Emerging Multi-Agent Intelligence with Economic Interactions
An economy of agents using auctions and wealth accumulation produces emergent multi-step reasoning that outperforms monolithic baselines on five agentic tasks.
-
AgensFlow: A Coordination-Policy Substrate for Multi-Agent Systems
AgensFlow learns coordination policies from task trajectories and outperforms fixed pipelines on distributed-systems incident and security-advisory tasks.
-
LATTICE: Evaluating Decision Support Utility of Crypto Agents
LATTICE is a scalable LLM-judge benchmark for crypto agent decision support that reveals performance trade-offs among real-world copilots across dimensions and tasks.
-
The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications
LLMs show low sycophancy to rebuttals but high sycophancy to conflicting user preferences in financial agentic tasks, with recovery methods benchmarked.
-
The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications
LLMs show low sycophancy to direct contradictions in financial tasks but high sycophancy to user preference contradictions, with input filtering as one recovery approach.
-
Complete Cyclic Subtask Graphs for Tool-Using LLM Agents: Flexibility, Cost, and Bottlenecks in Multi-Agent Workflows
Complete cyclic subtask graphs offer a lens to measure when multi-agent revisitation aids recovery and exploration versus when it increases costs or is dominated by other bottlenecks in LLM agent workflows.
-
BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows
BankerToolBench is a new open benchmark of end-to-end investment banking workflows developed with 502 bankers; even the best tested model (GPT-5.4) fails nearly half the expert rubric criteria and produces zero client...
-
FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use
FinToolBench couples 760 executable financial APIs with 295 tool-required queries and scores agents on execution success plus timeliness, intent, and domain compliance, with a finance-aware retrieval baseline (FATR).
-
Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows
Finch is a new benchmark with 172 composite workflows and 384 tasks from real enterprise data that shows top AI models like GPT-5.1 Pro pass only 38.4% of workflows under human evaluation.
-
Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application
This survey categorizes agentic environments for LLMs by eight attributes and domains, introduces symbolic and neural synthesis paradigms with evaluation, and outlines four agent evolution pathways plus three environm...
-
Absorbing Complexity: An Interaction-Native Knowledge Harness for Financial LLM Agents
InKH architecture absorbs complexity into financial LLM agents, cutting latency 83%, token cost 82%, and stale knowledge 97% while raising task quality 0.108 on a 46k-episode synthetic benchmark versus baselines.
-
AI Trading: Evaluating Large Language Models for Technical Market Analysis
A comparative evaluation claims GPT-4 Turbo and FinGPT outperformed the S&P 500 in a 2023 simulated backtest, but flawed baselines and missing code/data undermine the result.
-
Leakage-Aware Benchmarking of LLM Forecasting: Real-Time Nowcasts as the Decision-Time Input for Macro Factor Ranking
Leakage-controlled LLM factor ranking yields median Spearman IC of +0.154 that is largely matched by a kNN baseline on the same real-time macro inputs.
-
FinCom: A Financial Multi-Agent Demo with Disagree-or-Commit Deliberation
FinCom introduces a governed multi-agent system with Disagree-or-Commit deliberation that claims to improve reasoning accuracy and risk awareness over consensus baselines in financial tasks.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.