pith. sign in

arxiv: 2605.15482 · v1 · pith:TM7BM77Knew · submitted 2026-05-14 · 💻 cs.CL

FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models

Dmitry Stanishevskii , Nini Kamkia , Alexey Khoroshilov , Dmitry Zmitrovich , Denis Kokosinskii , Zhirayr Hayrapetyan , Andrei Kalmykov This is my paper

Pith reviewed 2026-05-19 14:27 UTC · model grok-4.3

classification 💻 cs.CL
keywords financial benchmarkslarge language modelshierarchical evaluationCFA certificationtechnical analysistrading tasksLLM evaluationfinancial reasoning
0
0 comments X p. Extension
pith:TM7BM77K Add to your LaTeX paper What is a Pith Number? 
\usepackage{pith}
\pithnumber{TM7BM77K}

Prints a linked pith:TM7BM77K badge after your title and writes the identifier into PDF metadata. Compiles on arXiv with no extra files. Learn more

The pith

FINESSE-Bench supplies 3993 questions in eight levels to test financial knowledge progression in large language models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces FINESSE-Bench as a collection of eight benchmarks that organize financial questions into a clear difficulty hierarchy. The questions come from certification-style materials, trading exercises, and a Russian olympiad set to check how models handle basic concepts before advancing to expert analysis and decisions. This structure reveals where performance drops as tasks grow harder and supports consistent scoring across answer formats. A sympathetic reader would care because prior financial benchmarks lack this step-by-step ladder, so it remains unclear whether models can move from simple queries to professional-grade work.

Core claim

We present FINESSE-Bench, a suite of eight specialized benchmarks comprising 3,993 questions for hierarchical evaluation of financial competencies in LLMs. FINESSE-Bench combines exam-oriented datasets inspired by professional certifications (CFA-like Levels 1-3, CMT-like Level 2, and CFTe-like Level 1), applied trading task collections, and a Russian-language olympiad benchmark. This design enables evaluation of domain breadth, performance degradation as difficulty increases, the ability to solve computational tasks, and model behavior in specialized financial domains.

What carries the argument

A hierarchy of eight benchmarks drawn from certification levels and trading tasks that measures how model accuracy changes with rising financial reasoning difficulty.

If this is right

  • Performance can be tracked for steady results or clear drops as questions move from basic to advanced financial topics.
  • The suite covers multiple answer formats with one protocol for multiple-choice, numerical, and short open-ended responses.
  • Applied trading tasks allow direct testing of computational financial skills.
  • Both English certification-style questions and a Russian-language set broaden coverage of specialized domains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The levels could guide targeted fine-tuning to close specific gaps between basic and expert financial reasoning.
  • Direct comparison of model scores to human pass rates on similar certifications would give practical context for the results.
  • Adding live market feeds to the trading tasks might expose limits not visible in static exam questions.
  • The multilingual element points to possible extensions for testing financial reasoning across more languages and markets.

Load-bearing premise

Questions modeled on certification exams and trading tasks form a genuine progression that matches how financial expertise develops in practice.

What would settle it

Model accuracy staying roughly equal across all eight levels, or benchmark scores showing no relation to success on real financial analysis jobs, would show the claimed hierarchy does not track actual competency growth.

Figures

Figures reproduced from arXiv: 2605.15482 by Alexey Khoroshilov, Andrei Kalmykov, Denis Kokosinskii, Dmitry Stanishevskii, Dmitry Zmitrovich, Nini Kamkia, Zhirayr Hayrapetyan.

Figure 1
Figure 1. Figure 1: Comparison of transfer gaps from classical open financial benchmarks to FINESSE [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Within-family scaling results for reasoning-oriented models from the Qwen3 family. [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
read the original abstract

Large language models (LLMs) are increasingly being applied to financial analysis, reporting, investment decision support, risk management, compliance, and professional training. However, robust evaluation of their domain competence in finance remains incomplete. Widely used open benchmarks such as FinQA, ConvFinQA, and TAT-QA have played an important role in advancing financial question answering and numerical reasoning, but they focus primarily on question answering over financial reports and do not provide an explicit hierarchy of professional difficulty. Broader resources, including FinanceBench, PIXIU, FinBen, and FLaME, expand the coverage of financial tasks, yet the problem of evaluating the transition from foundational knowledge to expert-level financial reasoning remains open. In this work, we present FINESSE-Bench, a suite of eight specialized benchmarks comprising 3,993 questions for hierarchical evaluation of financial competencies in LLMs. FINESSE-Bench combines exam-oriented datasets inspired by professional certifications (CFA-like Levels 1-3, CMT-like Level 2, and CFTe-like Level 1), applied trading task collections, and a Russian-language olympiad benchmark. This design enables evaluation of domain breadth, performance degradation as difficulty increases, the ability to solve computational tasks, and model behavior in specialized financial domains. We also describe a unified evaluation protocol covering multiple-choice questions, numerical answers, and short open-ended responses, together with an automated scoring scheme for freeform answers based on the LLM-as-judge paradigm. FINESSE-Bench is intended both as a complement to existing open financial benchmarks and as a tool for more substantive evaluation of professionally relevant financial competencies in large language models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces FINESSE-Bench, a suite of eight specialized benchmarks comprising 3,993 questions for hierarchical evaluation of financial competencies in LLMs. It combines exam-oriented datasets inspired by professional certifications (CFA-like Levels 1-3, CMT-like Level 2, CFTe-like Level 1), applied trading task collections, and a Russian-language olympiad benchmark. The work also describes a unified evaluation protocol for multiple-choice, numerical, and short open-ended responses, including an automated LLM-as-judge scoring scheme for freeform answers. The design aims to assess domain breadth, performance degradation with increasing difficulty, computational task solving, and behavior in specialized financial domains, positioning the benchmark as a complement to existing resources like FinQA and FinanceBench.

Significance. If the hierarchy is properly grounded, FINESSE-Bench would offer a useful addition to financial LLM evaluation by enabling structured assessment of the transition from foundational knowledge to expert-level reasoning, including performance degradation across levels and coverage of trading tasks and multilingual content. This addresses a noted gap in prior benchmarks that focus mainly on report-based QA without explicit professional difficulty progressions. The scale (3,993 questions) and multi-format protocol are practical strengths for reproducibility and broad applicability.

major comments (2)
  1. [Benchmark construction and dataset description sections] The central claim that the eight benchmarks form a valid progressive hierarchy for measuring performance degradation with expertise level is load-bearing but lacks supporting validation. The datasets are constructed as 'inspired by' certifications rather than directly sourced or calibrated against them. No details appear on the question generation process, expert review for accuracy and difficulty, pilot testing to confirm monotonic difficulty increase, or statistical checks (e.g., expert performance showing L1 systematically easier than L3 or trading tasks). Without this, observed LLM score patterns cannot be confidently attributed to hierarchical reasoning transitions versus dataset artifacts or inconsistent quality.
  2. [Evaluation protocol and scoring scheme] The evaluation protocol relies on an LLM-as-judge for scoring open-ended and freeform answers, yet no validation metrics are reported, such as agreement rates with human experts, correlation coefficients, or accuracy on a held-out set. This is critical because unreliable judging directly affects the trustworthiness of all numerical results and comparisons across the hierarchy.
minor comments (2)
  1. [Introduction] The abstract and introduction reference specific prior benchmarks (FinQA, ConvFinQA, TAT-QA, FinanceBench, PIXIU, FinBen, FLaME) but do not include a dedicated related-work table or explicit comparison of coverage gaps that FINESSE-Bench fills.
  2. [Benchmark overview] The total question count (3,993) is stated, but a breakdown table by benchmark and question type (MCQ vs. numerical vs. open-ended) would improve clarity on the distribution across difficulty levels.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and the recommendation of major revision. We address each major comment below, agreeing that additional details and validation are needed to strengthen the claims. We will revise the manuscript accordingly to improve clarity on benchmark construction and evaluation reliability.

read point-by-point responses

  1. Referee: [Benchmark construction and dataset description sections] The central claim that the eight benchmarks form a valid progressive hierarchy for measuring performance degradation with expertise level is load-bearing but lacks supporting validation. The datasets are constructed as 'inspired by' certifications rather than directly sourced or calibrated against them. No details appear on the question generation process, expert review for accuracy and difficulty, pilot testing to confirm monotonic difficulty increase, or statistical checks (e.g., expert performance showing L1 systematically easier than L3 or trading tasks). Without this, observed LLM score patterns cannot be confidently attributed to hierarchical reasoning transitions versus dataset artifacts or inconsistent quality.

    Authors: We agree that the manuscript requires expanded details to substantiate the hierarchical structure. In the revised version, we will add a new subsection under Benchmark Construction detailing the question generation process, which drew from publicly available CFA, CMT, and CFTe exam outlines, sample questions, and topic lists to create 'inspired by' items rather than direct copies. We will describe the expert review process, including involvement of financial domain specialists who verified factual accuracy, assigned difficulty levels aligned with certification standards, and ensured coverage of professional competencies. Pilot testing results with initial model runs will be reported to demonstrate observed performance degradation trends. For statistical checks, we will include analysis of LLM score patterns across levels (e.g., average accuracy decreasing from L1 to L3) as supporting evidence for monotonic difficulty, while explicitly noting that direct expert human performance data on these exact questions was not collected, as the items are newly synthesized. This clarification will help distinguish hierarchical effects from potential artifacts and ground the design in established certification progressions. revision: yes

  2. Referee: [Evaluation protocol and scoring scheme] The evaluation protocol relies on an LLM-as-judge for scoring open-ended and freeform answers, yet no validation metrics are reported, such as agreement rates with human experts, correlation coefficients, or accuracy on a held-out set. This is critical because unreliable judging directly affects the trustworthiness of all numerical results and comparisons across the hierarchy.

    Authors: We acknowledge the critical need for validation of the LLM-as-judge component. In the revised manuscript, we will expand the Evaluation Protocol section to report validation metrics on the scoring scheme. Specifically, we will include inter-rater agreement rates (e.g., Cohen's kappa or percentage agreement) between the LLM judge and human financial experts on a sampled subset of open-ended responses. We will also add correlation coefficients with human scores and accuracy metrics on a held-out validation set where available from our internal checks. If resource limitations prevented comprehensive human validation in the original experiments, we will discuss this transparently as a limitation and propose it as future work. These additions will directly address concerns about the reliability of numerical results and cross-hierarchy comparisons. revision: yes

Circularity Check

0 steps flagged

No circularity: benchmark construction paper with no derivations or fitted predictions

full rationale

The paper presents FINESSE-Bench as a new suite of 3,993 questions organized into eight benchmarks inspired by CFA, CMT, and CFTe certifications plus trading tasks. No mathematical derivations, equations, parameter fitting, or predictions appear in the provided text. The hierarchy is a design choice based on external certification structures rather than a self-referential reduction or self-citation chain. No load-bearing steps reduce claims to inputs by construction, satisfying the criteria for an honest non-finding of circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on the assumption that certification-style questions and trading tasks constitute a meaningful difficulty hierarchy and that LLM-as-judge scoring is adequate for open responses; no free parameters or new entities are introduced.

axioms (2)
  • domain assumption Professional certification exams (CFA, CMT, CFTe) provide a valid proxy for progressive financial competency levels
    The hierarchy is built directly on these exam structures.
  • domain assumption LLM-as-judge paradigm produces reliable scores for short open-ended financial answers
    The paper adopts this method for automated scoring of freeform responses.

pith-pipeline@v0.9.0 · 5866 in / 1414 out tokens · 68726 ms · 2026-05-19T14:27:38.639245+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 5 internal anchors

  1. [1]

    FinQA: A dataset of numerical reasoning over financial data

    Wenhu Chen Zhiyu Chen et al. FinQA: A dataset of numerical reasoning over financial data. arXiv preprint arXiv:2109.00122, 2021. URL https://arxiv.org/abs/2109.00122

    work page arXiv 2021

  2. [2]

    ConvFinQA: Exploring the chain of numerical reasoning in 20 conversational finance question answering.arXiv preprint arXiv:2210.03849, 2022

    Shiyang Li Zhiyu Chen et al. ConvFinQA: Exploring the chain of numerical reasoning in 20 conversational finance question answering.arXiv preprint arXiv:2210.03849, 2022. URL https://arxiv.org/abs/2210.03849

    work page arXiv 2022

  3. [3]

    TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance.arXiv preprint arXiv:2105.07624, 2021

    Wenqiang Lei Fengbin Zhu et al. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance.arXiv preprint arXiv:2105.07624, 2021. URL https://arxiv.org/abs/2105.07624

    work page arXiv 2021

  4. [4]

    FinanceBench: A New Benchmark for Financial Question Answering

    Anand Kannappan Pranab Islam et al. FinanceBench: A new benchmark for financial question answering.arXiv preprint arXiv:2311.11944, 2023. URL https://arxiv.org/abs/ 2311.11944

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    PIXIU: A large language model, instruction data and evaluation benchmark for finance.arXiv preprint arXiv:2306.05443, 2023

    Weiguang Han Qianqian Xie et al. PIXIU: A large language model, instruction data and evaluation benchmark for finance.arXiv preprint arXiv:2306.05443, 2023. URL https://arxiv.org/abs/2306.05443

    work page arXiv 2023

  6. [6]

    FinBen: A holistic financial benchmark for large language models

    Weiguang Han Qianqian Xie et al. FinBen: A holistic financial benchmark for large language models.arXiv preprint arXiv:2402.12659, 2024. URL https://arxiv.org/abs/2402.12659

    work page arXiv 2024

  7. [7]

    Finance language model evaluation (FLaME).arXiv preprint arXiv:2506.15846, 2025

    Mika Okamoto Glenn Matlin et al. Finance language model evaluation (FLaME).arXiv preprint arXiv:2506.15846, 2025. URL https://arxiv.org/abs/2506.15846

    work page arXiv 2025

  8. [8]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Wei-Lin Chiang Lianmin Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.arXiv preprint arXiv:2306.05685, 2023. URL https://arxiv.org/abs/2306.05685

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

    Wei-Lin Chiang Tianle Li et al. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939, 2024. URL https://arxiv.org/abs/2406.11939

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Financereasoning: Benchmarking financial numerical reasoning more credible, comprehensive and challenging.arXiv preprint arXiv:2506.05828, 2025

    Haihong E Zichen Tang et al. Financereasoning: Benchmarking financial numerical reasoning more credible, comprehensive and challenging.arXiv preprint arXiv:2506.05828, 2025. URL https://arxiv.org/abs/2506.05828

    work page arXiv 2025

  11. [11]

    Fin-R1: A large language model for financial reasoning through reinforcement learning.arXiv preprint arXiv:2503.16252, 2025

    Xin Guo Zhaowei Liu et al. Fin-R1: A large language model for financial reasoning through reinforcement learning.arXiv preprint arXiv:2503.16252, 2025. URL https://arxiv.org/abs/ 2503.16252

    work page arXiv 2025

  12. [12]

    arena-hard-auto

    LMArena. arena-hard-auto. https://github.com/lmarena/arena-hard-auto. GitHub reposi- tory, accessed April 2026

    work page 2026

  13. [13]

    Measuring Massive Multitask Language Understanding

    Collin Burns Dan Hendrycks et al. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2021. URL https://arxiv.org/abs/2009.03300

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  14. [14]

    CMMLU: Measuring massive multitask language understanding in Chinese

    Yixuan Zhang Haonan Li et al. CMMLU: Measuring massive multitask language under- standing in chinese.arXiv preprint arXiv:2306.09212, 2024. URL https://arxiv.org/abs/ 2306.09212

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [15]

    Reasoning models are test exploiters: Rethinking multiple choice.arXiv preprint arXiv:2507.15337, 2025

    Kevin Leyton-Brown Narun Raman, Taylor Lundy. Reasoning models are test exploiters: Rethinking multiple choice.arXiv preprint arXiv:2507.15337, 2025. URL https://arxiv.org/ html/2507.15337v1. 21

    work page arXiv 2025