REVIEW 4 major objections 9 minor 1 cited by
FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language Models
T0 review · 4 major / 9 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read FinanceQA is a benchmark built from real analyst work; its central claim is that leading LLMs fail about 60% of realistic financial analysis tasks, with the best model, o1, at 48.7%.
desk verdict FinanceQA is a genuinely useful new benchmark for job-realistic finance LLM evaluation, but its headline failure rates need a human baseline and per-category N before they can be taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the FinanceQA question-answer pair with an expert-written gold answer. Each tactical question carries a context window assembled only from relevant 10-K sections, forcing models to recalculate from primary sources rather than rely on memory; assumption-based questions intentionally omit a datum and require the model to state and apply a plausible assumption, such as scaling variable lease costs to estimate variable lease assets. Questions are scored as exact matches, so a single missed adjustment, such as failing to add back operating lease costs in an EBITDA calculation, fails the item. The fine-tuning arm uses synthetic variations of human-written rows, 9,078 total, that preserve the reasoning template while randomizing line items and numbers, so the model learns the procedure rather than memorizing an answer. The benchmark's design is question-first, mirroring how analysts start with a question and then search documents, in contrast to context-first benchmarks that write questions from facts already present in the text.
What would settle it
Build a held-out set of tactical FinanceQA-style questions from 10-Ks of five companies in different industries, such as healthcare, energy, technology, industrials, and retail, and run the same four models with the same exact-match scoring. If accuracy on this multi-company set is materially higher than 48.7% overall, or well above 5% on assumption-based questions, then the paper's central failure-rate claim is an artifact of the single-company test set rather than a general property of LLMs in finance.
Extended reading notes
Core claim
FinanceQA is a testing suite of tactical and conceptual questions modeled on real analyst tasks. Tactical questions are grounded in a company's 10-K, with a context window made from the relevant sections, and split into 'basic' questions and 'assumption-based' questions that require an assumption the filing does not supply; conceptual questions test financial reasoning without a document. Evaluating GPT-4o, o1, Claude-3.5-Sonnet, and Llama-3.3-70B-Instruct with exact-match scoring, the paper finds total accuracies of 39.2%, 48.7%, 39.9%, and 31.1%, respectively, and assumption-question accuracies at or below 4.3%. It interprets the pattern as a specific capability gap: models handle textbook-style conceptual reasoning relatively well but fail precision recalculations, adherence to accounting conventions, and assumption generation under incomplete information. The paper then fine-tunes GPT-4o on synthetic variations of human-written financial reasoning pairs, producing a 44.9% relative improvement in total accuracy and a 690.9% relative improvement on assumption-based questions, with a paired t-test p-value of 0.034. The stated conclusion is that higher-quality, work-aligned training data is the missing ingredient, and that FinanceQA provides a way to measure progress toward it.
Load-bearing premise
The reported failure rates assume that tactical questions built from one company's 10-K (Costco) represent the full range of real-world financial analysis across industries, company types, and reporting styles.
Editorial extensions
If this is right
- Financial institutions cannot adopt current LLM output as work product: even a model scoring 80% would require line-by-line human verification that takes longer than doing the analysis from scratch.
- Assumption-based reasoning is the largest measured gap: with all models below 5%, any useful financial LLM must be trained and evaluated specifically on incomplete-information tasks.
- Fine-tuning on synthetic, convention-compliant reasoning data can substantially improve performance on real-world financial tasks, including a 44.9% relative total accuracy gain for GPT-4o.
- Benchmarks that start from facts in documents and ask for extraction overstate LLM ability; question-first benchmarks like FinanceQA measure the tasks professionals actually perform.
Reading between the lines
- An obvious extension the authors do not run is a multi-company, multi-industry FinanceQA, which would show whether the near-universal failure on assumption questions is a structural LLM limitation or partly a Costco-specific reporting artifact.
- The synthetic fine-tuning recipe, human-written templates with randomized numbers and line items, suggests a cheap scaling path; if the effect generalizes, increasing synthetic diversity could push accuracy past the materiality threshold, but the current 15.2% assumption accuracy is still far too low for unsupervised use.
- The same question-first benchmark methodology could transfer to other precision professions, such as legal drafting or clinical documentation, by extracting work-critical tasks from primary documents and expert answers.
- The paper's discontinuity argument implies that small accuracy gains around the 50-60% range are commercially meaningless until a much higher threshold is crossed; a useful next test would be measuring where human analysts stop verifying every line.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FinanceQA, a benchmark for evaluating LLMs on financial analysis tasks that the authors claim mirror real-world work by junior investment professionals. The benchmark consists of 'tactical' questions derived from a single company's 10-K (Costco), segmented into basic and assumption-based types, plus 'conceptual' questions. The authors evaluate GPT-4o, o1, Claude-3.5-Sonnet, and Llama-3.3-70B-Instruct, reporting low accuracies, especially on assumption-based questions (2.2–4.3%). They then fine-tune GPT-4o on a synthetic dataset derived from human-written finance questions and report substantial accuracy improvements (39.2% to 56.8% total), along with a paired t-test they claim shows statistical significance. The paper's central claim is that current LLMs fail the strict accuracy requirements of financial institutions, and that fine-tuning on high-quality financial reasoning data can improve performance.
Significance. If the benchmark is well-constructed and the results are reliable, FinanceQA could be a useful contribution to the evaluation of LLMs in professional finance, a domain where existing benchmarks such as FinanceBench are criticized for being too easy. The paper also ships a publicly released dataset and demonstrates a fine-tuning pipeline, which are praiseworthy. However, the current evidence is insufficient to support the strong claims: the benchmark lacks published dataset sizes, confidence intervals, a human expert baseline, and a statistically sound analysis of the fine-tuning gains. The qualitative direction (LLMs struggle with exact-match financial calculation under incomplete information) is plausible, but the quantitative claims need substantial strengthening.
major comments (4)
- [§5.2, Table 1] The paper never reports the number of questions in each category (basic, assumption, conceptual) or in total. Without per-category N, the percentages in Table 1 are uninterpretable—for example, the assumption-based accuracies of 2.2–4.3% could represent a single question difference (1/45 vs. 1/23) or a larger effect. This omission also precludes confidence intervals and makes it impossible to evaluate the stability of the central claim that models 'fail less than 5%' on assumption-based questions. Please report exact question counts for every cell, and ideally provide binomial confidence intervals or exact per-question results.
- [§6.2, Figure 7] The paired t-test is conducted on only three paired aggregates (basic, assumption, conceptual), giving df = 2. With n = 3, the test cannot meaningfully assess statistical significance: the assumptions of the t-test (normality and independence) are unverifiable, and the three observations are not independent outcomes but aggregated proportions from the same benchmark. Moreover, the test ignores question-level variation entirely. The reported p-value of 0.034 therefore does not support the claim that the fine-tuning improvements are 'statistically significant at the 95% confidence level.' Please reanalyze at the question level (e.g., a paired permutation test or a mixed-effects model across individual questions), or clearly limit the significance claim to the three category aggregates and explain why a t-test with df=2 is appropriate.
- [§5.2, Table 1; §4] No expert-human baseline is provided under the same exact-match rubric. The paper's central interpretation—that low LLM scores indicate a failure to meet 'strict accuracy requirements of financial institutions'—requires knowing how well trained analysts perform on the same questions with the same grading rules. The concern is especially acute for assumption-based questions, where the expected answer depends on an author-chosen assumption (e.g., the variable-lease ratio in Section 4). Different but defensible assumptions would yield different final numbers and would be marked wrong under exact match. Without a human baseline, the low LLM scores could reflect grading strictness rather than an LLM-specific deficit. Please add at least a small-scale human expert evaluation under identical conditions, or substantially temper the claim that the results demonstrate an LLM-specific failure.
- [§4, §8] All tactical questions are based on a single company's 10-K (Costco), as the authors acknowledge in Section 8. The benchmark's external validity is therefore extremely narrow: Costco's financial structure is not representative of healthcare, energy, financial services, or other industries, and a single company cannot capture the diversity of 'real-world tasks' the abstract claims to evaluate. The paper should either restrict its generalization claims to the Costco-like retail sector or include additional companies before using the benchmark to draw broad conclusions about LLM performance in professional finance.
minor comments (9)
- [Abstract] The claim that 'models fail approximately 60% of realistic tasks' conflicts with the best model's performance: o1 fails about 51.3% (accuracy 48.7% per Table 1). The 60% figure appears to be an average across the four models; please state this explicitly or use a more accurate summary.
- [§4.1, Dataset Annotation] The paper states that annotators manually verified questions and answers to be unambiguous, but no inter-annotator agreement statistics are reported. Please add a measure of agreement (e.g., Cohen's kappa on a subset) or at least a detailed description of the double-checking procedure.
- [§5.3] The sentence 'unless directly prompted, it won’t follow know that it needs to calculate those metrics' contains a typo ('follow know' should be 'know' or 'follow through').
- [§3.4] The section heading 'Handing Incomplete Information' should be 'Handling Incomplete Information'.
- [§2] The phrase 'makes the model suspect to the same dilemma' is awkward; 'susceptible' would be more natural.
- [Figure 4] The correlation matrix figure is not described in the text; it is unclear which variables are correlated (presumably accuracies across question types). Please explain the figure in a sentence and provide the numeric correlation values if possible.
- [§6.1] The fine-tuning dataset consists of 9,078 rows, but the paper does not report how many unique human-written seeds were used, how many synthetic variations were generated per seed, or the ratio of tactical to conceptual questions. Please clarify the dataset composition.
- [§5.1, Setup] The specific model versions are not fully identified (e.g., o1-preview vs. o1-2024-12-17, and the exact release dates for GPT-4o, Claude-3.5-Sonnet, and Llama-3.3-70B-Instruct). Please provide version identifiers and access dates for reproducibility.
- [Figure 7] What is labeled 'Figure 7' is in fact a table of percentages and percent differences; please renumber or relabel it as a table.
Circularity Check
No significant circularity: the benchmark results and fine-tuning evaluation are empirical measurements, not derivations that reduce to their inputs.
full rationale
FinanceQA is an empirical benchmark paper rather than a derivation: the central claims are measurements on a constructed dataset, so the usual circular-derivation patterns (fitted input called prediction, self-citation importing uniqueness, ansatz smuggled via citation) do not apply. The authors build tactical questions from primary documents (Costco 10-K) and conceptual questions from practitioner experience; annotator-set expected answers and exact-match grading are transparently stated in Sections 4 and 5.1. The fine-tuning experiment (Section 6) is the only place where training and evaluation data are related: the training rows are described as independent of the evaluation set ('These new questions were designed to be independent of the evaluation set, including different industries and problems'), and the evaluation still uses FinanceQA. That is a standard benchmark fine-tuning design, not a by-construction identity, and no fitted parameter is renamed as a prediction. The main weaknesses—no expert-human baseline under the same exact-match rubric, no per-category sample sizes, and a single-company tactical set (Section 8)—undermine calibration and generalization, but they are validity limitations, not circularity. No load-bearing self-citations appear in the reference list; citations to textbooks and standards are external sources. Accordingly no circular step is identified.
Assumptions & free parameters
assumptions (4)
- domain assumption Annotator-created answers are the correct and unambiguous ground truth.
- domain assumption Exact-match binary scoring captures the professional quality of an answer.
- ad hoc to paper Synthetic data variations preserve the reasoning quality of human-written seeds.
- ad hoc to paper Paired differences are normally distributed in the significance test.
Cite this review
Pith. "Pith review of FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language Models." pith.science (2026). https://pith.science/paper/NCGV6TNO
@misc{pith2026250118062,
author = {Pith},
title = {Pith review of: FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/NCGV6TNO}},
note = {Machine review of arXiv:2501.18062}
}
read the original abstract
FinanceQA is a testing suite that evaluates LLMs' performance on complex numerical financial analysis tasks that mirror real-world investment work. Despite recent advances, current LLMs fail to meet the strict accuracy requirements of financial institutions, with models failing approximately 60% of realistic tasks that mimic on-the-job analyses at hedge funds, private equity firms, investment banks, and other financial institutions. The primary challenges include hand-spreading metrics, adhering to standard accounting and corporate valuation conventions, and performing analysis under incomplete information - particularly in multi-step tasks requiring assumption generation. This performance gap highlights the disconnect between existing LLM capabilities and the demands of professional financial analysis that are inadequately tested by current testing architectures. Results show that higher-quality training data is needed to support such tasks, which we experiment with using OpenAI's fine-tuning API. FinanceQA is publicly released at [this https URL](https://huggingface.co/datasets/AfterQuery/FinanceQA).
Forward citations
Cited by 1 Pith paper
-
On Path to Multimodal Historical Reasoning: HistBench and HistAgent
HistAgent, a history-specialized agent, scores 27.54% pass@1 and 36.47% pass@2 on the new 414-question HistBench benchmark, surpassing generalist agents tested on the same data.
Reference graph
Works this paper leans on
-
[1]
Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Langdon, D., ... & Wang, W. Y. (2021). Finqa: A dataset of numerical reasoning over financial data. arXiv preprint arXiv:2109.00122. Guha, N., Nyarko, J., He, D., Re, C., Chilton, A., Narayanan, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D.N., Zambrano, D., Talisman, D., Hoque, E., Surani,...
arXiv 2021
-
[3]
arXiv preprint arXiv:2311.11944 Koller, T., Goedhart, M., & Wessels, D
FINANCEBENCH: A New Benchmark for Financial Question Answering. arXiv preprint arXiv:2311.11944 Koller, T., Goedhart, M., & Wessels, D. (2020). Valuation: Measuring and Managing the Value of Companies, 7th edition . John Wiley & Sons Inc. http://hdl.handle.net/1765/125836 Matton, A., Sherborne, T., Aumiller, D., Tommasone, E., Alizadeh, M., He, J., Ma, R....
arXiv 2020
-
[99]
(1999). Materiality. Securities and Exchange Commission . https://www.sec.gov/interps/account/sab99.htm CFA Institute. (2024). Introduction to financial statement analysis. Retrieved from https://www.cfainstitute.org/insights/profes sional-learning/refresher-readings/2024/introduction-financial-statement-analysis S&P Global Ratings. (2024). Leveraged fina...
arXiv 1999
-
[2023]
arXiv preprint arXiv:2308.11462
LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. arXiv preprint arXiv:2308.11462. Ernst & Young. (2021). How the leases standard impacts company balance sheets. Retrieved from https://www.ey.com/en_us/insights/ifrs/how-the-leases-standard-impacts-company-balance-sheets Holthausen, R. W., & Zmijewski, M....
arXiv 2021
-
[2024]
arXiv preprint arXiv:2407.07565
On Leakage of Code Generation Evaluation Datasets. arXiv preprint arXiv:2407.07565. SEC Staff Accounting Bulletin No
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.