WorkstreamBench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance
Pith reviewed 2026-05-22 05:25 UTC · model grok-4.3
The pith
Current LLM agents fall short of professional standards when creating complete financial spreadsheets from high-level instructions.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
LLM agents are not yet able to reliably produce professional-quality spreadsheets at the level of complexity real-world financial workflows demand, as shown by their sharp performance degradation on harder tasks despite leading models producing the most professional-looking outputs.
What carries the argument
A three-dimensional evaluation taxonomy with Accuracy, Formula, and Format dimensions, each containing fine-grained criteria that reflect professional finance standards for reviewed deliverables.
If this is right
- If agents can close this gap, they could automate substantial portions of financial modeling and analysis in enterprise settings.
- Performance degrades with increasing difficulty, implying that current models lack robust capabilities for long sequences of dependent calculations.
- Qualitative review shows Claude family outputs look most professional, suggesting differences in how models handle structure and presentation.
- The benchmark focuses on end-to-end tasks, which could guide development toward more practical AI tools for finance.
Where Pith is reading between the lines
- Such a benchmark might be extended to other domains like accounting or data analysis where spreadsheets are central.
- Improvements here could lead to AI assistants that not only compute but also create maintainable documents for teams.
- Tracking progress on this benchmark over time could measure advancement in agentic AI for professional tasks.
Load-bearing premise
The assumption that the Accuracy, Formula, and Format dimensions together capture the key criteria that finance stakeholders apply when they review and revise spreadsheet work.
What would settle it
A controlled study in which finance professionals rate the quality of agent-generated spreadsheets on real tasks and check whether high benchmark scores align with their professional approval or rejection.
Figures
read the original abstract
LLM agents are increasingly expected to carry out end-to-end workflows, producing complete artifacts from high-level user instructions. To meet enterprise needs, frontier AI labs have developed agents that can construct entire spreadsheets from scratch. This is especially relevant in finance, where core workflows such as financial modeling, forecasting, and scenario analysis are commonly conducted through spreadsheets. Yet, existing spreadsheet benchmarks do not measure this advanced capability, focusing instead on question-answering or single-formula edits. To address this gap, we provide one of the first evaluations of agents on end-to-end spreadsheet tasks, focusing on economically critical financial workflows such as modeling and scenario analysis. Since deliverables therein are routinely reviewed and revised by multiple stakeholders, judging their quality necessarily involves high-level criteria such as readability or ease of modification. To reflect the multidimensional nature of solution quality, we develop an evaluation taxonomy comprising three dimensions: Accuracy, Formula, and Format, each comprising fine-grained criteria that reflect professional standards. The Claude family leads the benchmark and produces the most professional-looking outputs in our qualitative review, but even the strongest agents frequently fall short of professional finance standards and degrade sharply as the difficulty increases beyond a few chained calculations. This suggests that current agents are not yet able to reliably produce professional-quality spreadsheets at the level of complexity real-world workflows demand.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces WorkstreamBench, one of the first benchmarks for evaluating LLM agents on end-to-end spreadsheet construction tasks in finance, such as financial modeling, forecasting, and scenario analysis. It defines a three-dimensional evaluation taxonomy (Accuracy, Formula, Format) with fine-grained criteria intended to reflect professional standards, evaluates frontier agents on these tasks, and reports that the Claude family produces the most professional-looking outputs while all agents frequently fall short of professional finance standards and degrade sharply with increasing difficulty beyond a few chained calculations.
Significance. If the taxonomy is shown to align with actual finance-professional review practices, the benchmark fills a clear gap left by prior spreadsheet evaluations focused on question-answering or single-formula edits. The work supplies concrete evidence of current agent limitations on complex, multi-stakeholder deliverables and offers a reusable evaluation framework that could steer development toward more reliable enterprise agents. The qualitative observation that Claude outputs appear most professional is a useful secondary finding.
major comments (1)
- Abstract: The central claim that agents 'frequently fall short of professional finance standards' rests on the assertion that the Accuracy/Formula/Format taxonomy 'reflect professional standards.' No evidence is provided of external validation—such as expert inter-rater agreement, correlation with real revision rates, or blind comparison against existing finance review rubrics—which is load-bearing for interpreting the measured performance gaps and difficulty scaling as evidence of professional inadequacy.
minor comments (1)
- Abstract: The headline results would be easier to contextualize if the number of tasks, difficulty levels, and agents evaluated were stated explicitly rather than left implicit.
Simulated Author's Rebuttal
We thank the referee for their constructive feedback, which helps clarify the scope and limitations of our evaluation framework. We address the major comment point by point below.
read point-by-point responses
-
Referee: Abstract: The central claim that agents 'frequently fall short of professional finance standards' rests on the assertion that the Accuracy/Formula/Format taxonomy 'reflect professional standards.' No evidence is provided of external validation—such as expert inter-rater agreement, correlation with real revision rates, or blind comparison against existing finance review rubrics—which is load-bearing for interpreting the measured performance gaps and difficulty scaling as evidence of professional inadequacy.
Authors: We agree that the manuscript would benefit from clearer justification of how the taxonomy aligns with professional practice. The three dimensions and their criteria were derived from widely cited guidelines in financial modeling (e.g., best-practice recommendations from the CFA Institute, Wall Street training materials on model auditability, and common review criteria used in investment banking for model hand-off). However, we did not perform formal external validation such as inter-rater reliability studies with practicing finance professionals or correlation analyses against real-world revision rates. In the revised version we will (1) expand the Methods section with explicit references to the professional sources used to construct each criterion, (2) add a dedicated Limitations subsection that acknowledges the absence of direct expert validation and the interpretive caution this implies, and (3) revise the abstract and conclusion to state that agents fall short of the standards encoded in the taxonomy rather than asserting professional inadequacy without qualification. These changes will make the evidential basis for our claims more transparent. revision: yes
Circularity Check
No significant circularity in derivation chain
full rationale
This is an empirical benchmark paper with no mathematical derivations, equations, or fitted parameters. The evaluation taxonomy is introduced as a methodological framework whose criteria are stated to reflect professional standards; this is a definitional choice for measurement rather than a prediction or result derived from prior inputs by construction. Central claims consist of direct performance measurements and qualitative observations against the stated criteria. No self-citation chains or uniqueness theorems are invoked as load-bearing premises. The work is self-contained as an empirical study, consistent with the default expectation of no circularity for such evaluations.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption The selected tasks (financial modeling, forecasting, scenario analysis) are representative of real-world finance workflows that require multi-stakeholder review.
- domain assumption The Accuracy-Formula-Format taxonomy reflects the criteria actually used by finance professionals when judging spreadsheet deliverables.
invented entities (1)
-
WorkstreamBench benchmark
no independent evidence
Lean theorems connected to this paper
-
IndisputableMonolith/Foundation/RealityFromDistinction.leanreality_from_one_distinction unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
develop an evaluation taxonomy comprising three dimensions: Accuracy, Formula, and Format, each comprising fine-grained criteria that reflect professional standards
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Reference graph
Works this paper leans on
-
[1]
Anthropic. Use Claude for Excel. https://support.claude.com/en/articles/ 12650343-use-claude-for-excel, 2025a. Accessed: 2026-05-01. OpenAI. ChatGPT for Excel and Google Sheets in beta. https://chatgpt.com/apps/ spreadsheets/, 2026a. Accessed: 2026-05-01. CFA Institute. Financial Modeling. https://www.cfainstitute.org/ programs/cfa-program/candidate-resou...
work page 2026
-
[2]
Association for Financial Professionals
Accessed: 2026-05-01. Association for Financial Professionals. 2025 AFP FP&A Benchmarking Survey Report: Technology & Data. https://www.financialprofessionals.org/ training-resources/resources/survey-research-economic-data/Details/ afp-fpa-benchmarking-survey-report-technology,
work page 2026
-
[3]
Accessed: 2026-05-01. Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volum...
work page 2026
-
[4]
doi: 10.52202/079017-3007. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ ac840df270ac537dd74530a15c332684-Paper-Datasets_and_Benchmarks_Track.pdf. Yibin Chen, Yifu Yuan, Zeyu Zhang, Yan Zheng, Jinyi Liu, Fei Ni, Jianye Hao, Hangyu Mao, and Fuzheng Zhang. SheetAgent: Towards A Generalist Agent for Spreadsheet Reasoning and Manipulation vi...
-
[5]
NL2Formula: Generating spreadsheet formulas from natural lan- guage queries
Wei Zhao, Zhitao Hou, Siyuan Wu, Yan Gao, Haoyu Dong, Yao Wan, Hongyu Zhang, Yulei Sui, and Haidong Zhang. NL2Formula: Generating spreadsheet formulas from natural lan- guage queries. In Yvette Graham and Matthew Purver, editors,Findings of the Association for Computational Linguistics: EACL 2024, pages 2377–2388, St. Julian’s, Malta, March
work page 2024
-
[6]
doi: 10.18653/v1/2024.findings-eacl.158
Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-eacl.158. URL https://aclanthology.org/2024.findings-eacl.158/. Haoyu Dong, Pengkun Zhang, Yan Gao, Xuanyu Dong, Yilin Cheng, Mingzhe Lu, Zikun Zhu, Adina Yakefu, and Shuxin Zheng. Finch: Benchmarking finance & accounting across spreadsheet-centric enterprise workflows,
-
[7]
Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows
URLhttps://arxiv.org/abs/2512.13168. 10 Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, and Jerry Tworek. GDPval: Eval...
work page internal anchor Pith review Pith/arXiv arXiv
-
[8]
URLhttps://arxiv.org/abs/2510.04374. Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, Zhiyang Deng, Yuechen Jiang, Zhiyuan Yao, Haohang Li, Yangyang Yu, Gang Hu, ...
work page internal anchor Pith review Pith/arXiv arXiv
-
[9]
Curran Associates Inc. ISBN 9798331314385. Zhihan Zhang, Yixin Cao, and Lizi Liao. XFinBench: Benchmarking LLMs in complex finan- cial problem solving and reasoning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Findings of the Association for Computational Lin- guistics: ACL 2025, pages 8715–8758, Vienna, Austria, July
work page 2025
-
[10]
Association for Computa- tional Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.457. URL https://aclanthology.org/2025.findings-acl.457/. Michael Krumdick, Rik Koncel-Kedziorski, Viet Dac Lai, Varshini Reddy, Charles Lovering, and Chris Tanner. BizBench: A quantitative reasoning benchmark for business and finance. In Lun- Wei Ku, A...
-
[11]
In: Zong, C., Xia, F., Li, W., Navigli, R
Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long.452. URLhttps://aclanthology.org/2024.acl-long.452/. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. PIXIU: A comprehensive benchmark, instruction dataset and large language model for finance. InThirty-seventh Conference on Neur...
-
[12]
Guilong Lu, Xuntao Guo, Rongjunchen Zhang, Wenqiao Zhu, and Ji Liu
URLhttps://openreview.net/forum?id=vTrRq6vCQH. Guilong Lu, Xuntao Guo, Rongjunchen Zhang, Wenqiao Zhu, and Ji Liu. Bizfinbench: A business- driven real-world financial benchmark for evaluating llms.arXiv preprint arXiv:2505.19457,
-
[13]
History of Financial Modeling Competitions
FMWC. History of Financial Modeling Competitions. https://fmworldcup.com/history/, 2026a. Accessed: 2026-05-01. FMWC. Financial Modeling World Cup. https://fmworldcup.com/, 2026b. Accessed: 2026- 05-01. WSP. Wall Street Prep.https://www.wsp.com/en-us/,
work page 2026
-
[14]
Accessed: 2026-05-01. OpenAI. Introducing ChatGPT Agent. https://openai.com/index/ introducing-chatgpt-agent/,
work page 2026
-
[15]
Accessed: 2026-01-29. Microsoft Corporation. Playwright: A framework for Web Testing and Automation,
work page 2026
-
[16]
Accessed: 2026-05-06. OpenAI. ChatGPT.https://chatgpt.com/, 2026b. Accessed: 2026-05-01. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can Language Models Resolve Real-world Github Is- sues? InThe Twelfth International Conference on Learning Representations,
work page 2026
-
[17]
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
URL https://arxiv.org/abs/2405.15793. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854,
work page internal anchor Pith review Pith/arXiv arXiv
-
[18]
Anthropic. Claude Opus 4.5 System Card. https://assets.anthropic.com/m/ 64823ba7485345a7/Claude-Opus-4-5-System-Card.pdf, 2025b. Accessed: 2026-01-29. A Evaluation Criteria — Sub-dimensions In this section, we provide more examples for the rubric’s subdimensions to illustrate what they capture. A.1 Using Absolute References Consider converting product pri...
work page 2026
-
[19]
B.6 Task Difficulty Classification — Exemplary Tasks Exemplary Tasks by Difficulty Level Illustrative competition tasks for each difficulty level Very Easy — Task 7 (Accounts Receivable) A single-step task with direct formula application, requiring no finance knowledge and no Excel skills. Easy — Task 380 (First Time Buyer) Involves only a few steps with ...
work page 2015
-
[20]
xhigh",max_completion_tokens=128000 • Claude Opus 4.6: anthropic_effort=
Claude (Excel)An official Anthropic add-in available through the Microsoft Marketplace that embeds a Claude-powered sidebar directly inside Excel for Windows, Mac, and Excel on the web. Users install it via the Excel Add-ins menu, sign in with a qualifying Anthropic account (Pro, Max, Team, or Enterprise), and interact with Claude through a chat panel wit...
work page 2025
-
[21]
is supported by our qualitative investigations. One example of agents’ poorer performance is the change in the font color of the labels. For instance, in one of ChatGPT (Excel)’s attempts, the agent changed the font color in two consecutive rows (see Figure 15). For a client-ready model, this lack of consistency in presentation would be unacceptable. Figu...
work page 2019
-
[22]
This requires reasoning about the spatial structure of the workbook and whether the freeze pane is placed to actually serve its navigational purpose, not simply checking whether one is present. "decision":"Fail", "summary":"The freeze pane on the ’Model’ sheet is set incorrectly, hindering readability as timeline headers do not remain visible when scrolli...
work page 2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.