Finance Language Model Evaluation (FLaME)

Glenn Matlin; Huzaifa Pardawala; Mika Okamoto; Sudheer Chava; Yang Yang

REVIEW 3 major objections 6 minor 25 references

Finance Language Model Evaluation (FLaME)

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FLaME claims to be the first holistic benchmarking suite for financial language model evaluation, measuring 23 foundation LMs on 20 finance NLP tasks.

desk verdict A genuinely reusable FinNLP benchmark suite whose comparative rankings are weaker than they look because all models share one prompt set tuned on Llama 3 8B. read the letter →

arxiv 2506.15846 v1 pith:QJGQYKS3 submitted 2025-06-18 cs.CL cs.AIcs.CE

Glenn Matlin , Mika Okamoto , Huzaifa Pardawala , Yang Yang , Sudheer Chava This is my paper · ORCID

classification cs.CLcs.AIcs.CE

keywords financialNLPlanguagemodelevaluationholisticbenchmarkreasoning-reinforcedmodelszero-shotpromptingtaxonomypublicleaderboardcost-performancetrade-off

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that existing financial-language benchmarks understate what language models can do because they lack standardized pipelines, multi-metric scoring, and an explicit account of what they leave out. To close that gap it presents FLaME, which it calls the first holistic benchmarking suite for core finance NLP tasks: 23 foundation models, including reasoning-reinforced ones, evaluated on 20 datasets under a common pipeline. The suite adds a scenario taxonomy, a public leaderboard, and open code, data, and results, so others can reproduce the numbers and extend the benchmark. If the claim holds, the field gains a shared yardstick for comparing models on financial text tasks.

What carries the argument

The carrying object is the FLaME evaluation pipeline: a unified inference layer that sends the same prompts to local or cloud models, a separate extraction model that pulls structured answers out of free-form generations, and task-specific metrics including an LM-as-judge for numerical QA. Around this pipeline sits a scenario taxonomy that classifies every dataset by task, domain, and language, which the paper uses to expose coverage gaps and to justify its claim of holistic coverage. The whole system runs under the three criteria the paper adopts for 'holistic': standardized evaluation, multi-metric assessment, and explicit recognition of incompleteness.

What would settle it

Run the same 20 tasks on a handful of models (for example DeepSeek R1, Claude 3.5 Sonnet, GPT-4o, and Llama 3 8B) with per-model tuned prompts or a few-shot variant, and compare the task-level rankings against the paper's zero-shot rankings; if the top models change materially, the paper's comparative conclusions are prompt-dependent.

Watch

Extended reading notes

Core claim

On its own terms, the paper's finding is that a single standardized, multi-metric evaluation of current foundation LMs reveals a more capable picture of finance NLP than earlier suites suggested, while showing that no one model dominates. Reasoning-reinforced models, meaning models trained to produce extended step-by-step reasoning before answering, such as DeepSeek R1, lead multi-step and conversational question answering; Claude 3.5 Sonnet leads sentiment and some relation-extraction tasks; and GPT-4o hovers near the top in classification and summarization. Meanwhile open-weight mid-scale models such as DeepSeek-V3 and Llama 3.1 70B deliver comparable performance at a fraction of the inference cost. Numeric labeling (FNXL) and causal analysis remain hard for every model, and summarization is comparatively tractable. The paper interprets these results as evidence that task and domain structure, not raw scale, drive performance in finance NLP.

Load-bearing premise

The comparative rankings assume that a single set of zero-shot prompts, tuned on one model and then applied unchanged to 23 models, measures all models fairly; if prompt sensitivity differs across models, the rankings could be an artifact of prompt compatibility rather than of capability.

Editorial extensions

If this is right

No single foundation model wins across all 20 tasks, so practitioners should choose models per task rather than by overall score.
Reasoning-reinforced models dominate multi-step conversational QA but cost several times more; for many tasks open-weight mid-scale models offer a better accuracy-per-dollar trade-off.
Numeric labeling and causal reasoning in financial text remain unsolved by zero-shot prompting, pointing to areas where domain-adaptive training is needed.
The public leaderboard and open software make the benchmark a living resource: new models or datasets can be added without rebuilding the pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the paper tunes its zero-shot prompts on a single model and applies them unchanged to all others, the headline rankings could partly reflect prompt compatibility rather than underlying ability; per-model prompt adaptation might reorder the leaderboard.
The taxonomy's explicit mapping of data gaps could double as a research agenda: the emptiest cells (non-English, causal, decision-making scenarios) are the places where new dataset creation would add the most information.
If the 'first holistic' label depends on the three adopted criteria, rival suites could adopt the same criteria and the first-mover status would fade; the durable contribution would then be the open pipeline rather than the priority claim.
A testable extension is whether fine-tuned smaller models on each task beat the best zero-shot foundation models; the paper's cost analysis suggests this could be a cheaper path for real deployments.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. First, this is the rare benchmark paper that actually ships: code, data, results, leaderboard, and a cost table. That part is solid. Second, do not take the model rankings at face value. Every model was probed with the same zero-shot prompts, tuned on Llama 3 8B, and the paper itself admits in Section F.2 that this may not generalize. It then documents the exact ways it doesn't: Banking77 label variants, Qwen drifting into Chinese, FinQA numeric formatting. Under those conditions, the headline results (DeepSeek R1 beating everyone at conversational QA, Claude 3.5 winning sentiment) are as much about prompt compatibility as about capability.

What's genuinely new: a 23-model, 20-dataset evaluation with a scenario taxonomy grounded in HELM, plus an explicit incompleteness discussion and a public leaderboard. The taxonomy is a useful way to see which financial scenarios have no data, and the cost analysis is a real contribution. The error analysis tables are honest and specific. That's more than most benchmark papers provide.

The soft spots are proportional. The prompt issue is real and central to the comparative half of the claims. The 'first holistic' framing is overused—the abstract says 20 core tasks but it's 20 datasets across 6 task categories, and the multi-metric claim is uneven since some datasets get a single accuracy number. Four datasets (FiNER-ORD, FOMC, NumClaim, SubjECTive-QA) come from the authors' own lab; that's a selection-bias flag, not a smoking gun, but it should be disclosed more prominently. There are also no error bars or repeated runs, though temperature 0 makes that less critical.

Who gets value: anyone building or evaluating financial NLP models who wants a standardized harness and a gap map. The paper deserves a serious referee: it's a substantial infrastructure artifact and the flaws are fixable with language softening and a per-model prompt sensitivity check. I'd take it for review, but I'd ask the authors to either bring the prompt variation evidence into the main text or explicitly reframe the leaderboard as a starting point, not a verdict.

Referee Report

3 major / 6 minor

Summary. This paper presents FLaME, an open-source benchmark suite and evaluation pipeline for financial NLP. It proposes a scenario taxonomy over tasks, domains, and languages; curates 20 datasets under explicit inclusion criteria spanning six task categories (information retrieval, sentiment analysis, causal analysis, text classification, question answering, summarization); evaluates 23 foundation language models, including three called 'reasoning-reinforced' models, under zero-shot deterministic decoding; and reports per-dataset metrics, an efficiency/cost analysis, a public leaderboard, and an error analysis. The central claims are that FLaME is the first holistic (HELM-style) financial benchmark—satisfying standardization, recognition of incompleteness, and multi-metric evaluation—and that the paper provides the first comprehensive empirical comparison of standard versus reasoning-reinforced LMs, with headline findings such as DeepSeek R1 leading multi-hop QA and Claude 3.5 Sonnet leading FPB sentiment.

Significance. If the claims hold, FLaME is a valuable community asset: it ships code, data, results, and prompts; documents dataset licenses, selection criteria, and provenance; provides a taxonomy that makes coverage gaps explicit; and publishes an unusually concrete error analysis. The transparency of the pipeline (prompts in Appendix C.4, costs in Table 13, per-task tables in Appendix F) is a genuine strength, and the explicit recognition of incompleteness in Section 6 and Appendix H follows HELM practice. The empirical rankings are interesting but their strength depends on prompt robustness, and the artifact value does not depend on those rankings; the infrastructure and the public leaderboard are likely to be useful to the FinNLP community regardless of the specific model ordering.

major comments (3)

[§4.1, §F.2] The central comparative claim (the rankings and the standard-versus-reasoning-reinforced comparison) rests on a single zero-shot prompt set tuned on Llama 3 8B and applied verbatim to all 23 models. The paper itself concedes in F.2 that this 'may not fully generalize to others' and documents exactly the per-model format failures that can bias comparisons: Banking77 label syntax variants, Qwen 2 drifting into Chinese, and FinQA '34.81%' versus '34.8%' mismatches. Because the headline findings (e.g., DeepSeek R1 at 0.853 on ConvFinQA versus 0.840 for o1-mini and 0.749 for GPT-4o, §F.3.5) could be prompt-compatibility artifacts rather than capability differences, the paper should either run a prompt-sensitivity analysis (two or three prompt variants per task on a subset of models, reporting rank stability) or perform per-model prompt validation, and in either case soften the specific 'leads' and 'dominates' language in §4.1 and §F.3. I consider this load-bearing because the empirical comparison is half of the paper's central claim.
[Abstract, §3.3, Table 2] The abstract's '20 core NLP tasks' conflates the 20 dataset/task columns in Table 2 with the six task categories defined in §3.2 and with the 20 datasets claimed in §3.3 (CD and CC are two subtasks of FinCausal-SC, and FinEntity is counted in two categories). Contribution 2 also says '20 financial benchmarks,' so the tasks/datasets/benchmarks terminology is inconsistent across the abstract, the contributions, and the body. Relatedly, the 'multi-metric' pillar of the holistic claim is weaker than presented: Table 2 reports a single headline metric per dataset (F1, accuracy, MSE, or BERTScore), and the additional precision/recall/F1 columns in Appendix F are standard per-task metrics rather than the complementary cross-scenario metric families that HELM-style multi-metric evaluation contemplates. The paper should either implement a uniform multi-metric protocol or reframe the holistic claim accordingly.
[§4.1, §F.2, §F.3.5] The paper flags in F.2 that FinQA, TATQA, and EDTSum may overlap with model pre-training corpora and that this 'may inflate' zero-shot scores, yet the headline conclusion 'DeepSeek R1 dominates multi-step QA' rests precisely on FinQA, ConvFinQA, and TATQA. The paper also reports a single deterministic run per model with no confidence intervals or significance tests while presenting small gaps as rankings (e.g., FinQA 0.844 for Claude 3.5 Sonnet versus 0.840 for DeepSeek-V3 versus 0.836 for GPT-4o and DeepSeek R1). These comparative conclusions should either be accompanied by uncertainty quantification, including finite-test-set standard errors, or the 'leads/dominates' language should be scaled back to 'achieves the highest observed score in this protocol.'

minor comments (6)

[§3.4] The Evaluation section describes a configurable weighting system and a final meta-score that 'accounts for the relative performance range of models across tasks,' but no meta-score is computed or reported anywhere in Section 4 or the appendices; either report it or remove the description.
[Table 2 caption, §1] The set of 'reasoning-reinforced' models is never defined even though comparing standard versus reasoning-reinforced LMs is stated as a central contribution; the caption of Table 2 bolds three models without stating the criterion, so the reader cannot tell whether the category is defined by training procedure, provider self-description, or observed reasoning behavior.
[§4.3, Table 13, Table 4] Model naming is inconsistent: §4.3 and Table 13 refer to 'Llama 3.1 8b' while Tables 2 and 4 use 'Llama 3 8B,' and Table 4 lists Llama 2 13B Chat even though F.2 excludes it from the 23 models in the main results; the model inventory should be reconciled.
[Table 1, Appendix G] The comparative checkmarks in Table 1 are categorical and largely unaudited; in particular, FinBen (Xie et al., 2024), credited with 22 datasets and 9 models, is marked down on multiple criteria without detailed justification, and Appendix G's rebuttal is assertive rather than evidential. Since the 'first holistic benchmark' claim is partly a comparative claim, the criteria and their application need to be reproducible.
[Table 10, §4.1] The header 'Casual Classification' in Table 10 should read 'Causal Classification,' and the all-caps 'CONVFINQA' in §4.1 is inconsistent with 'ConvFinQA' elsewhere; there are also scattered typos such as 'major limiting favor' in Section 6.
[Contributions, Ethics Statement] The contributions section contains a liability disclaimer (that co-authors 'should not be held liable for errors') which is out of place in the paper body, and the Ethics Statement says that details on the leaderboard and visualizations 'will be updated for publication,' a draft artifact that should be resolved before submission.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: FLaME's results are measured, not derived; the only self-referential element (four datasets from the authors' own lab) is not load-bearing in any derivation sense.

full rationale

FLaME is an empirical benchmark suite, not a derivation chain. There are no equations, fitted parameters, or constructed predictions that reduce to their own inputs. The central claims — the holistic evaluation framework and the model rankings — are produced by direct measurement (inference plus metric computation), so there is no fitted-input-called-prediction pattern. The paper's own admission in Section F.2 that prompts were tuned on Llama 3 8B 'for cost reasons' and 'may not fully generalize to others' is a prompt-compatibility threat to the cross-model rankings, but it is a validity concern, not circularity: the reported DeepSeek R1 or Claude 3.5 results are not enforced by the tuning procedure. The only self-referential element is that four of the twenty datasets (FiNER-ORD, FOMC, NumClaim, SubjECTive-QA) originate from the same GT FinTech lab as the present paper; however, each was published independently with its own external annotation and the paper does not use these datasets as justification for any methodological claim. The 'first holistic' novelty claim is self-assessed against the HELM criteria, but it is a comparative bibliographic assertion, not a derivation that presupposes its conclusion. No uniqueness theorem, no ansatz-importing citation, and no renaming of a known result as unification appears. Accordingly, the paper merits a low circularity score of 1.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on measurement assumptions rather than mathematical derivation. There are no free parameters or invented entities; the main axioms are the HELM-based definition of holistic, the representativeness of the 20 datasets, the fairness of single zero-shot prompts, and the stability of deterministic decoding.

assumptions (4)

domain assumption The HELM definition of holistic evaluation (standardization, recognition of incompleteness, multi-metric evaluation) is the correct standard for judging FinNLP benchmarks.
Adopted in Section 1 from Liang et al. (2022); the 'first holistic' claim depends entirely on accepting these three pillars and on FLaME satisfying them.
domain assumption Zero-shot prompting with the provided prompt templates measures foundation LM capability on core FinNLP tasks.
Stated in Sections 3.4 and 6; the results are only meaningful if prompt sensitivity does not dominate model differences, but Section F.2 admits prompts were tuned on Llama 3 8B.
domain assumption The 20 selected datasets adequately represent the core financial NLP task space defined by the taxonomy.
Section 3.3 applies inclusion criteria such as domain relevance, licensing, and annotation quality; if the selection is unrepresentative, the benchmark's conclusions about model capability in finance are incomplete.
domain assumption Deterministic decoding with temperature 0, top-p 0.9, and one output per instance gives stable measurements.
Section 3.4 Generation; the paper reports no variance or repeated runs, though cloud APIs may not be perfectly deterministic.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Finance Language Model Evaluation (FLaME)." pith.science (2026). https://pith.science/paper/QJGQYKS3

@misc{pith2026250615846,
  author       = {Pith},
  title        = {Pith review of: Finance Language Model Evaluation (FLaME)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QJGQYKS3}},
  note         = {Machine review of arXiv:2506.15846}
}

read the original abstract

Language Models (LMs) have demonstrated impressive capabilities with core Natural Language Processing (NLP) tasks. The effectiveness of LMs for highly specialized knowledge-intensive tasks in finance remains difficult to assess due to major gaps in the methodologies of existing evaluation frameworks, which have caused an erroneous belief in a far lower bound of LMs' performance on common Finance NLP (FinNLP) tasks. To demonstrate the potential of LMs for these FinNLP tasks, we present the first holistic benchmarking suite for Financial Language Model Evaluation (FLaME). We are the first research paper to comprehensively study LMs against 'reasoning-reinforced' LMs, with an empirical study of 23 foundation LMs over 20 core NLP tasks in finance. We open-source our framework software along with all data and results.

Figures

Figures reproduced from arXiv: 2506.15846 by the authors.

**Figure 1.** Technical Overview: FLAME uses a unified inference hub, providing a single, model-agnostic API across three deployment modes: (i) proprietary cloud APIs (e.g., Claude 4, Gemini 2.5 Pro), (ii) cloud-hosted open-weight models (e.g., OLMo 2, Qwen 2.5) served by either cloud providers (TogetherAI, HuggingFace), and (iii) fully local inference backends (e.g., vLLM, Ollama). This modular software design abstracts deployme… view at source ↗

**Figure 2.** Functional Overview: In the Execution phase, a language model (e.g., DeepSeek-r1) generates responses to financial queries. During the Evaluation phase, text spans are extracted from the generated text by an LM (e.g., Llama3 3B), followed by either directly verifying the answer or using automated evaluation performed by a judge LM (e.g., Llama3 8B). FLAME’s main contribution is providing a comprehensive software pac… view at source ↗

**Figure 3.** Illustrative breakdown for each of the six core NLP task categories. While our taxonomy groups these tasks broadly, each category can encompass numerous specialized variants depending on data format, user needs, and domain constraints. We provide a limited set of specific examples to illustrate the concepts. generated, how it was created, and why it exists. Domains include financial institutions, regulators, news … view at source ↗

Figures from the paper (17 more)

**Figure 4.** Figure 4: Holistic Taxonomy for FLAME. Unlike prior FinNLP benchmark suites, which primarily collect individual datasets aligned to specific tasks or metrics, FLAME adopts a holistic perspective, systematically mapping benchmarks across multiple dimensions such as tasks, scenari…

**Figure 6.** Figure 6: Zero-shot prompt used for ConvFinQA. • TAT-QA (TQA) (Zhu et al., 2021) is a largescale question-answering (QA) dataset designed for hybrid data sources, combining both tabular and textual content, particularly from financial reports. The dataset emphasizes numerical…

**Figure 5.** Figure 5: Zero-shot prompt used for FinQA. • ConvFinQA (CFQA)(Chen et al., 2022) multi-turn question answering is a large-scale dataset designed to explore the chain of numerical reasoning in conversational questionanswering within the financial domain. It consists of 3,892 co…

**Figure 8.** Figure 8: Zero-shot prompt used for ECTSum. summaries. It comprises 2,000 financial news articles, each paired with its headline serving as the ground-truth summary. These articles were manually selected and cleaned from the dataset introduced by to ensure high-quality annotatio…

**Figure 9.** Figure 9: Zero-shot prompt used for EDTSum [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

**Figure 10.** Figure 10: Zero-shot prompt used for FiNER. • FinEntity (FE) (Tang et al., 2023) is an entity-level sentiment classification dataset designed for financial news analysis. It contains 979 financial news paragraphs, featuring 2,131 manually-annotated financial entities classifi…

**Figure 11.** Figure 11: Zero-shot prompt used for FinEntity. this task requires set-based numerical comparison. Thus, we cannot use Entity F1 scores directly. Normalization is applied consistently across all datasets to reduce inconsistencies, including case standardization and whitespace …

**Figure 13.** Figure 13: Zero-shot prompt used for FinRED. • REFinD (RD) (Kaur et al., 2023) is a specialized relation extraction dataset created to address the unique challenges of extracting relationships between entity pairs from financial texts. With approximately 29,000 annotated insta…

**Figure 12.** Figure 12: Zero-shot prompt used for FNXL. • FinRED (FR) (Sharma et al., 2022) dataset is a specialized relation extraction dataset tailored to the financial domain, created to address the gap where existing models trained on general datasets fail to transfer effectively to fi…

**Figure 14.** Figure 14: Zero-shot prompt used for ReFinD. Sentiment Analysis. • FiQA (Maia et al., 2018) has two sub tasks. FiQA Task 1 focuses on aspect-based financial sentiment analysis. Given a financial text, such as microblog posts or news headlines, systems are tasked with identifyin…

**Figure 17.** Figure 17: Zero-shot prompt used for SubjECTiveQA. Text Classification. • Banking77 (B77) (Casanueva et al., 2020) is a fine-grained dataset designed for intent detection within the banking domain. It comprises 13,083 customer service queries annotated with 77 unique intents, s…

**Figure 16.** Figure 16: Zero-shot prompt used for FPB. • SubjECTive-QA (SQA) (Pardawala et al., 2024) is a manually-annotated dataset focusing on subjectivity and soft misinformation in Earnings Call Transcripts (ECTs), specifically in their long-form QA sessions. It includes 49,446 annot…

**Figure 18.** Figure 18: Zero-shot prompt used for Banking77. FinBench dataset consists of approximately 333,000 labeled instances, covering three primary financial risks: default, fraud, and churn. Each instance is labeled as "high risk" or "low risk". The time frame of data collection vari…

**Figure 19.** Figure 19: Zero-shot prompt used for FinBench. This dataset facilitates the study of numerical claim detection in financial discourse and serves as a resource for argument mining and investor sentiment analysis. For evaluation, we prompted the language models to output if each …

**Figure 20.** Figure 20: Zero-shot prompt used for NumClaim. • News Headline (HL) Classification (Sinha and Khandait, 2021) dataset consists of 11,412 human-annotated financial news headlines focused on commodities, particularly [PITH_FULL_IMAGE:figures/full_fig_p029_20.png]

**Figure 22.** Figure 22: Zero-shot prompt used for FOMC. Causal Analysis. • FinCausal-SC (Mariko et al., 2020) is a dataset for cause-effect analysis in financial news texts. It consists of 29,444 text sections (each containing up to three sentences), with 2,136 annotated as causal and accomp…

**Figure 23.** Figure 23: Zero-shot prompt used for Causal Detection. [PITH_FULL_IMAGE:figures/full_fig_p031_23.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages

[1]

Identify every numeral in the sentence

work page
[2]

other". Return only valid JSON in this format: ```json {

For each numeral, assign the most appropriate US-GAAP XBRL tag based on context. If no tag is appropriate, label it as "other". Return only valid JSON in this format: ```json { "12.0": "us-gaap:Revenue", "9.5": "us-gaap:SomeExpense", "100.0": "other" }``` The sentence is: {sentence} """ Figure 12: Zero-shot prompt used for FNXL. • FinRED (FR)(Sharma et al...

work page 2022
[3]

How do rising interest rates influence credit default swaps?

They can require domain experts to judge the validity of reasoning steps, significantly in- creasing the cost of dataset creation and eval- uation. (2) Knowledge.Tasks such asfact completion, knowledge-intensive QA, andcritical reasoning are pivotal in scenarios requiring specialized fi- nancial intelligence. A language model might need to recall policy c...

work page
[4]

Haoqiang Kang and Xiao-Yang Liu

Challenges and applications of large language models.arXiv [cs.CL]. Haoqiang Kang and Xiao-Yang Liu. 2023. Deficiency of large language models in finance: An empirical examination of hallucination.arXiv [cs.CL]. Simerjot Kaur, Charese Smiley, Akshat Gupta, Joy Sain, Dongsheng Wang, Suchetha Siddagangappa, Toyin Aguda, and Sameena Shah. 2023. REFinD: Rela-...

work page 2023
[5]

Teven Le Scao and Alexander Rush

BizBench: A quantitative reasoning bench- mark for business and finance.arXiv [cs.CL]. Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Stroudsburg, PA, USA. Association for Comp...

work page 2021
[6]

differentiation

Numerical claim detection in finance: A new financial dataset, weak-supervision model, and mar- ket analysis.arXiv [cs.CL]. Agam Shah, Suvan Paturi, and Sudheer Chava. 2023a. Trillion dollar words: A new financial dataset, task & market analysis.arXiv [cs.CL]. Agam Shah, Ruchit Vithani, Abhinav Gullapalli, and Sudheer Chava. 2023b. FiNER: Financial named ...

work page 2023
[7]

What was Company X’s net income last quarter?

Sudheer Chavaprovided their expertise and feedback which provided valuable perspective throughout this project and helped refine our approach at key stages. A Taxonomy of Financial Scenarios Tasks.We focus on six core NLP tasks —ques- tion answering,information retrieval,summariza- tion,sentiment analysis,toxicity detection, andtext classificationcategory...

work page 1999
[11]

"" Discard all the previous instructions. Behave like you are an expert at fine-grained single-domain intent detection. From the following list: [

is a manually-annotated dataset focus- ing on subjectivity and soft misinformation in Earnings Call Transcripts (ECTs), specif- ically in their long-form QA sessions. It in- cludes 49,446 annotations across 2,747 QA pairs from 120 ECTs spanning 2007 to 2021. Each QA pair is labeled on six subjectiv- ity features: Assertive, Cautious, Optimistic, Specific,...

work page arXiv 2022

Show all 25 references

[12]

They often demand carefully labeled multi- step annotations (e.g., detailed solution out- lines for financial math problems)
[13]

They rely on domain-specific metrics that go well beyond typical F1 or BLEU scores (e.g., verifying the correctness of an interest-rate calculation, or confirming that code compiles and produces the right financial outputs)
[15]

Coverage:Maintaining an up-to-date repos- itory of financial facts (e.g., corporate struc- tures, compliance rules) is daunting due to constant changes in markets and regulatory environments
[16]

(3) Decision-Making.Finance ultimately re- volves around decision-making tasks such asmar- ket forecasting,risk management,stock-movement prediction, andcredit scoring

Verification and Fact-Checking:Complex financial facts often demand external refer- ences (e.g., official filings), and verifying cor- rectness is non-trivial. (3) Decision-Making.Finance ultimately re- volves around decision-making tasks such asmar- ket forecasting,risk manag...
[17]

Access to real-time or historicalstructured financial data (e.g., stock price feeds)
[18]

Well-defined metrics that can meaningfully assess predictive accuracy or risk-adjusted re- turns
[19]

(4) Human Alignment.Large language mod- els can inadvertently propagate harmful behav- iors—e.g., misinformation, social biases, or privacy violations

Potential integration of ethical and legal con- straints (e.g., insider trading regulations). (4) Human Alignment.Large language mod- els can inadvertently propagate harmful behav- iors—e.g., misinformation, social biases, or privacy violations. In finance, these concerns beco...
[20]

Social Bias and Toxicity: Minimizing harm- ful language and misinformation
[21]

Privacy and Copyright: Preventing models from disclosing proprietary or regulated infor- mation
[22]

(5) Multi-Modal.Many real financial workflows rely on data that is not purely text—e.g., Excel spreadsheets, visual charts, scanned PDF state- ments, or contract images

Regulatory Compliance: Evolving laws may require auditing an LLM’s data usage or out- put content. (5) Multi-Modal.Many real financial workflows rely on data that is not purely text—e.g., Excel spreadsheets, visual charts, scanned PDF state- ments, or contract images. Tasks li...
[23]

Specialized architectures or bridging modules that fuse text with tabular or image data
[24]

Domain-adapted evaluation methods (e.g., metrics for chart-based questions)
[25]

holiflame/dataset_name

Substantial cross-disciplinary expertise to an- notate or interpret financial images and tables consistently. As such, we limit FLAME to text-only tasks for its initial release, but we envision future expansions that incorporate multi-modal data sources in an end-to-end benchm...

2023
[2020]

Pranab Islam, Anand Kannappan, Douwe Kiela, Re- becca Qian, Nino Scherrer, and Bertie Vidgen

Measuring massive multitask language under- standing.arXiv [cs.CY]. Pranab Islam, Anand Kannappan, Douwe Kiela, Re- becca Qian, Nino Scherrer, and Bertie Vidgen. 2023. FinanceBench: A new benchmark for financial ques- tion answering.arXiv [cs.CL]. Ziwei Ji, Nayeon Lee, Rita Fr...

2023
[2021]

For evaluation, we prompted the language models to output a summary given an article, and compared that summary to the ground truth summary with BERTScore

focuses on corporate event detection and text-based stock prediction, containing 9,721 news articles with token-level event labels and 303,893 first-hand news articles with minute-level timestamps and comprehen- sive stock price labels. For evaluation, we prompted the language...
[2023]

Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang

A survey on evaluation of large language mod- els.arXiv [cs.CL]. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A dataset of nu- merical reasoning ...

2021
[2024]

arXiv [cs.CL]

A survey on large language models for criti- cal societal domains: Finance, healthcare, and law. arXiv [cs.CL]. Yew Ken Chia, Pengfei Hong, Lidong Bing, and Sou- janya Poria. 2023. INSTRUCTEV AL: Towards holis- tic evaluation of instruction-tuned large language models.arXiv [c...

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Identify every numeral in the sentence

work page

[2] [2]

other". Return only valid JSON in this format: ```json {

For each numeral, assign the most appropriate US-GAAP XBRL tag based on context. If no tag is appropriate, label it as "other". Return only valid JSON in this format: ```json { "12.0": "us-gaap:Revenue", "9.5": "us-gaap:SomeExpense", "100.0": "other" }``` The sentence is: {sentence} """ Figure 12: Zero-shot prompt used for FNXL. • FinRED (FR)(Sharma et al...

work page 2022

[3] [3]

How do rising interest rates influence credit default swaps?

They can require domain experts to judge the validity of reasoning steps, significantly in- creasing the cost of dataset creation and eval- uation. (2) Knowledge.Tasks such asfact completion, knowledge-intensive QA, andcritical reasoning are pivotal in scenarios requiring specialized fi- nancial intelligence. A language model might need to recall policy c...

work page

[4] [4]

Haoqiang Kang and Xiao-Yang Liu

Challenges and applications of large language models.arXiv [cs.CL]. Haoqiang Kang and Xiao-Yang Liu. 2023. Deficiency of large language models in finance: An empirical examination of hallucination.arXiv [cs.CL]. Simerjot Kaur, Charese Smiley, Akshat Gupta, Joy Sain, Dongsheng Wang, Suchetha Siddagangappa, Toyin Aguda, and Sameena Shah. 2023. REFinD: Rela-...

work page 2023

[5] [5]

Teven Le Scao and Alexander Rush

BizBench: A quantitative reasoning bench- mark for business and finance.arXiv [cs.CL]. Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Stroudsburg, PA, USA. Association for Comp...

work page 2021

[6] [6]

differentiation

Numerical claim detection in finance: A new financial dataset, weak-supervision model, and mar- ket analysis.arXiv [cs.CL]. Agam Shah, Suvan Paturi, and Sudheer Chava. 2023a. Trillion dollar words: A new financial dataset, task & market analysis.arXiv [cs.CL]. Agam Shah, Ruchit Vithani, Abhinav Gullapalli, and Sudheer Chava. 2023b. FiNER: Financial named ...

work page 2023

[7] [7]

What was Company X’s net income last quarter?

Sudheer Chavaprovided their expertise and feedback which provided valuable perspective throughout this project and helped refine our approach at key stages. A Taxonomy of Financial Scenarios Tasks.We focus on six core NLP tasks —ques- tion answering,information retrieval,summariza- tion,sentiment analysis,toxicity detection, andtext classificationcategory...

work page 1999

[8] [11]

"" Discard all the previous instructions. Behave like you are an expert at fine-grained single-domain intent detection. From the following list: [

is a manually-annotated dataset focus- ing on subjectivity and soft misinformation in Earnings Call Transcripts (ECTs), specif- ically in their long-form QA sessions. It in- cludes 49,446 annotations across 2,747 QA pairs from 120 ECTs spanning 2007 to 2021. Each QA pair is labeled on six subjectiv- ity features: Assertive, Cautious, Optimistic, Specific,...

work page arXiv 2022