{"id":"a8e17dd5-ef88-4b9c-83fd-aa22d18f60ed","arxiv_id":"2506.15846","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FLaME builds a standardized, open-source benchmark suite that evaluates 23 language models on 20 financial NLP datasets and reports performance, costs, and a scenario taxonomy.","lead":"FLaME is an open-source benchmark that runs 23 language models over 20 finance NLP datasets and publishes the code, data, and results. It gives researchers and practitioners a common yardstick for comparing models on financial text tasks, with a taxonomy, a public leaderboard, and cost figures.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparative rankings rest on prompts tuned on Llama 3 8B and applied verbatim to all 23 models; if prompt sensitivity varies by model, the headline DeepSeek R1 / Claude 3.5 results may be artifacts of prompt compatibility rather than capability.","rationale":"The reader's weakest assumption pinpoints the same load-bearing concern: the fixed prompt set tuned on Llama 3 8B creates an uncontrolled confound for every model-to-model comparison. I considered alternatives—the 'multi-metric' label, data contamination, and missing error bars—but the prompt-fairness issue is the most central because it directly determines whether the headline empirical study measures what it claims. It is also internally acknowledged in §F.2, so it is a correctness risk rather than an outside-consensus disagreement. The open-source software, dataset curation, and taxonomy retain independent value, which is why the appropriate action is a conditional revision rather than rejection. The proposed check is feasible with the released framework and would settle whether the reported rankings are stable under minimal, fair prompt variation.","tokens_in":54235,"tokens_out":5410,"duration_ms":65392,"concrete_test":"Select two contested comparisons from Table 2 (e.g., DeepSeek R1 vs. GPT-4o on Banking77 and on TAT-QA; Claude 3.5 vs. GPT-4o on FinQA). For each model, construct 5–10 prompt variants that differ only in instruction phrasing, label formatting, and answer extraction instructions, all without per-model test-set tuning. Run both the original and variant prompts on a held-out validation split of the same datasets, then compare rankings using the original test split. If the per-model best prompts flip the winner or change the top-3 ordering on more than one of the six task categories, the fixed-prompt leaderboard is not a reliable measure of relative LM capability and the empirical-comparison claim should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FLaME's central empirical claim—the first comprehensive comparison of standard and reasoning-reinforced LMs—requires that a single zero-shot prompt set, applied to all 23 models, yields comparable measurements. The paper admits in §F.2 that prompt tuning was done on Llama 3 8B for cost reasons and 'may not fully generalize to others,' and it documents the exact failure mode: Banking77 models emit syntax variants of labels, Qwen 2 drifts into Chinese, and FinQA numeric formatting ('34.81%' vs. '34.8%') is marked wrong. Because prompt compatibility is known to vary across model families, the reported rankings (DeepSeek R1 first in QA/IR, Claude 3.5 first in FPB, Jamba 1.5 Mini first in FinBench) cannot be separated from prompt compatibility. This does not challenge the released infrastructure or the taxonomy, but it does undercut the comparative conclusion that is half of the central claim. A per-model prompt check would settle whether the leaderboard reflects model capability or prompt fit.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents FLaME, an open-source benchmark suite and evaluation pipeline for financial NLP. It proposes a scenario taxonomy over tasks, domains, and languages; curates 20 datasets under explicit inclusion criteria spanning six task categories (information retrieval, sentiment analysis, causal analysis, text classification, question answering, summarization); evaluates 23 foundation language models, including three called 'reasoning-reinforced' models, under zero-shot deterministic decoding; and reports per-dataset metrics, an efficiency/cost analysis, a public leaderboard, and an error analysis. The central claims are that FLaME is the first holistic (HELM-style) financial benchmark—satisfying standardization, recognition of incompleteness, and multi-metric evaluation—and that the paper provides the first comprehensive empirical comparison of standard versus reasoning-reinforced LMs, with headline findings such as DeepSeek R1 leading multi-hop QA and Claude 3.5 Sonnet leading FPB sentiment.","tokens_in":54455,"tokens_out":12057,"duration_ms":126158,"significance":"If the claims hold, FLaME is a valuable community asset: it ships code, data, results, and prompts; documents dataset licenses, selection criteria, and provenance; provides a taxonomy that makes coverage gaps explicit; and publishes an unusually concrete error analysis. The transparency of the pipeline (prompts in Appendix C.4, costs in Table 13, per-task tables in Appendix F) is a genuine strength, and the explicit recognition of incompleteness in Section 6 and Appendix H follows HELM practice. The empirical rankings are interesting but their strength depends on prompt robustness, and the artifact value does not depend on those rankings; the infrastructure and the public leaderboard are likely to be useful to the FinNLP community regardless of the specific model ordering.","major_comments":[{"comment":"The central comparative claim (the rankings and the standard-versus-reasoning-reinforced comparison) rests on a single zero-shot prompt set tuned on Llama 3 8B and applied verbatim to all 23 models. The paper itself concedes in F.2 that this 'may not fully generalize to others' and documents exactly the per-model format failures that can bias comparisons: Banking77 label syntax variants, Qwen 2 drifting into Chinese, and FinQA '34.81%' versus '34.8%' mismatches. Because the headline findings (e.g., DeepSeek R1 at 0.853 on ConvFinQA versus 0.840 for o1-mini and 0.749 for GPT-4o, §F.3.5) could be prompt-compatibility artifacts rather than capability differences, the paper should either run a prompt-sensitivity analysis (two or three prompt variants per task on a subset of models, reporting rank stability) or perform per-model prompt validation, and in either case soften the specific 'leads' and 'dominates' language in §4.1 and §F.3. I consider this load-bearing because the empirical comparison is half of the paper's central claim.","section":"§4.1, §F.2"},{"comment":"The abstract's '20 core NLP tasks' conflates the 20 dataset/task columns in Table 2 with the six task categories defined in §3.2 and with the 20 datasets claimed in §3.3 (CD and CC are two subtasks of FinCausal-SC, and FinEntity is counted in two categories). Contribution 2 also says '20 financial benchmarks,' so the tasks/datasets/benchmarks terminology is inconsistent across the abstract, the contributions, and the body. Relatedly, the 'multi-metric' pillar of the holistic claim is weaker than presented: Table 2 reports a single headline metric per dataset (F1, accuracy, MSE, or BERTScore), and the additional precision/recall/F1 columns in Appendix F are standard per-task metrics rather than the complementary cross-scenario metric families that HELM-style multi-metric evaluation contemplates. The paper should either implement a uniform multi-metric protocol or reframe the holistic claim accordingly.","section":"Abstract, §3.3, Table 2"},{"comment":"The paper flags in F.2 that FinQA, TATQA, and EDTSum may overlap with model pre-training corpora and that this 'may inflate' zero-shot scores, yet the headline conclusion 'DeepSeek R1 dominates multi-step QA' rests precisely on FinQA, ConvFinQA, and TATQA. The paper also reports a single deterministic run per model with no confidence intervals or significance tests while presenting small gaps as rankings (e.g., FinQA 0.844 for Claude 3.5 Sonnet versus 0.840 for DeepSeek-V3 versus 0.836 for GPT-4o and DeepSeek R1). These comparative conclusions should either be accompanied by uncertainty quantification, including finite-test-set standard errors, or the 'leads/dominates' language should be scaled back to 'achieves the highest observed score in this protocol.'","section":"§4.1, §F.2, §F.3.5"}],"minor_comments":[{"comment":"The Evaluation section describes a configurable weighting system and a final meta-score that 'accounts for the relative performance range of models across tasks,' but no meta-score is computed or reported anywhere in Section 4 or the appendices; either report it or remove the description.","section":"§3.4"},{"comment":"The set of 'reasoning-reinforced' models is never defined even though comparing standard versus reasoning-reinforced LMs is stated as a central contribution; the caption of Table 2 bolds three models without stating the criterion, so the reader cannot tell whether the category is defined by training procedure, provider self-description, or observed reasoning behavior.","section":"Table 2 caption, §1"},{"comment":"Model naming is inconsistent: §4.3 and Table 13 refer to 'Llama 3.1 8b' while Tables 2 and 4 use 'Llama 3 8B,' and Table 4 lists Llama 2 13B Chat even though F.2 excludes it from the 23 models in the main results; the model inventory should be reconciled.","section":"§4.3, Table 13, Table 4"},{"comment":"The comparative checkmarks in Table 1 are categorical and largely unaudited; in particular, FinBen (Xie et al., 2024), credited with 22 datasets and 9 models, is marked down on multiple criteria without detailed justification, and Appendix G's rebuttal is assertive rather than evidential. Since the 'first holistic benchmark' claim is partly a comparative claim, the criteria and their application need to be reproducible.","section":"Table 1, Appendix G"},{"comment":"The header 'Casual Classification' in Table 10 should read 'Causal Classification,' and the all-caps 'CONVFINQA' in §4.1 is inconsistent with 'ConvFinQA' elsewhere; there are also scattered typos such as 'major limiting favor' in Section 6.","section":"Table 10, §4.1"},{"comment":"The contributions section contains a liability disclaimer (that co-authors 'should not be held liable for errors') which is out of place in the paper body, and the Ethics Statement says that details on the leaderboard and visualizations 'will be updated for publication,' a draft artifact that should be resolved before submission.","section":"Contributions, Ethics Statement"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid artifact paper, and the open-source framework, transparent prompts, and detailed error analysis are genuine contributions. My main reservation, reflected in the major comments, is that the comparative conclusions are presented more strongly than the prompt-tuning and contamination caveats warrant; this is fixable with additional experiments. I would also flag for the editor that the novelty framing is aggressive: FinBen (Xie et al., 2024) is a closely related prior effort, and the Table 1 criteria are self-defined, so the 'first holistic benchmark' claim should be scrutinized during revision. Additionally, four of the curated datasets (FiNER-ORD, FOMC, NumClaim, SubjECTive-QA) and the FLUE baseline in Table 1 originate from the authors' own lab; a brief statement of this relationship would improve transparency regarding potential task-selection bias."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is the rare benchmark paper that actually ships: code, data, results, leaderboard, and a cost table. That part is solid. Second, do not take the model rankings at face value. Every model was probed with the same zero-shot prompts, tuned on Llama 3 8B, and the paper itself admits in Section F.2 that this may not generalize. It then documents the exact ways it doesn't: Banking77 label variants, Qwen drifting into Chinese, FinQA numeric formatting. Under those conditions, the headline results (DeepSeek R1 beating everyone at conversational QA, Claude 3.5 winning sentiment) are as much about prompt compatibility as about capability.\n\nWhat's genuinely new: a 23-model, 20-dataset evaluation with a scenario taxonomy grounded in HELM, plus an explicit incompleteness discussion and a public leaderboard. The taxonomy is a useful way to see which financial scenarios have no data, and the cost analysis is a real contribution. The error analysis tables are honest and specific. That's more than most benchmark papers provide.\n\nThe soft spots are proportional. The prompt issue is real and central to the comparative half of the claims. The 'first holistic' framing is overused—the abstract says 20 core tasks but it's 20 datasets across 6 task categories, and the multi-metric claim is uneven since some datasets get a single accuracy number. Four datasets (FiNER-ORD, FOMC, NumClaim, SubjECTive-QA) come from the authors' own lab; that's a selection-bias flag, not a smoking gun, but it should be disclosed more prominently. There are also no error bars or repeated runs, though temperature 0 makes that less critical.\n\nWho gets value: anyone building or evaluating financial NLP models who wants a standardized harness and a gap map. The paper deserves a serious referee: it's a substantial infrastructure artifact and the flaws are fixable with language softening and a per-model prompt sensitivity check. I'd take it for review, but I'd ask the authors to either bring the prompt variation evidence into the main text or explicitly reframe the leaderboard as a starting point, not a verdict.","headline":"A genuinely reusable FinNLP benchmark suite whose comparative rankings are weaker than they look because all models share one prompt set tuned on Llama 3 8B.","tokens_in":54971,"tokens_out":2446,"would_cite":true,"duration_ms":28217,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FLaME claims to be the first holistic benchmarking suite for financial language model evaluation, measuring 23 foundation LMs on 20 finance NLP tasks.","keywords":["financial NLP","language model evaluation","holistic benchmark","reasoning-reinforced models","zero-shot prompting","taxonomy","public leaderboard","cost-performance trade-off"],"falsifier":"Run the same 20 tasks on a handful of models (for example DeepSeek R1, Claude 3.5 Sonnet, GPT-4o, and Llama 3 8B) with per-model tuned prompts or a few-shot variant, and compare the task-level rankings against the paper's zero-shot rankings; if the top models change materially, the paper's comparative conclusions are prompt-dependent.","tokens_in":54052,"feed_emoji":"📊","tokens_out":5719,"duration_ms":58257,"temperature":0.7,"pith_summary":"This paper claims that existing financial-language benchmarks understate what language models can do because they lack standardized pipelines, multi-metric scoring, and an explicit account of what they leave out. To close that gap it presents FLaME, which it calls the first holistic benchmarking suite for core finance NLP tasks: 23 foundation models, including reasoning-reinforced ones, evaluated on 20 datasets under a common pipeline. The suite adds a scenario taxonomy, a public leaderboard, and open code, data, and results, so others can reproduce the numbers and extend the benchmark. If the claim holds, the field gains a shared yardstick for comparing models on financial text tasks.","feed_headline":"One benchmark puts 23 language models through 20 finance tasks","feed_subtitle":"FLaME claims the first holistic finance NLP standard, with open code, data, and a public leaderboard.","key_machinery":"The carrying object is the FLaME evaluation pipeline: a unified inference layer that sends the same prompts to local or cloud models, a separate extraction model that pulls structured answers out of free-form generations, and task-specific metrics including an LM-as-judge for numerical QA. Around this pipeline sits a scenario taxonomy that classifies every dataset by task, domain, and language, which the paper uses to expose coverage gaps and to justify its claim of holistic coverage. The whole system runs under the three criteria the paper adopts for 'holistic': standardized evaluation, multi-metric assessment, and explicit recognition of incompleteness.","core_discovery":"On its own terms, the paper's finding is that a single standardized, multi-metric evaluation of current foundation LMs reveals a more capable picture of finance NLP than earlier suites suggested, while showing that no one model dominates. Reasoning-reinforced models, meaning models trained to produce extended step-by-step reasoning before answering, such as DeepSeek R1, lead multi-step and conversational question answering; Claude 3.5 Sonnet leads sentiment and some relation-extraction tasks; and GPT-4o hovers near the top in classification and summarization. Meanwhile open-weight mid-scale models such as DeepSeek-V3 and Llama 3.1 70B deliver comparable performance at a fraction of the inference cost. Numeric labeling (FNXL) and causal analysis remain hard for every model, and summarization is comparatively tractable. The paper interprets these results as evidence that task and domain structure, not raw scale, drive performance in finance NLP.","pith_inferences":["Because the paper tunes its zero-shot prompts on a single model and applies them unchanged to all others, the headline rankings could partly reflect prompt compatibility rather than underlying ability; per-model prompt adaptation might reorder the leaderboard.","The taxonomy's explicit mapping of data gaps could double as a research agenda: the emptiest cells (non-English, causal, decision-making scenarios) are the places where new dataset creation would add the most information.","If the 'first holistic' label depends on the three adopted criteria, rival suites could adopt the same criteria and the first-mover status would fade; the durable contribution would then be the open pipeline rather than the priority claim.","A testable extension is whether fine-tuned smaller models on each task beat the best zero-shot foundation models; the paper's cost analysis suggests this could be a cheaper path for real deployments."],"forward_implications":["No single foundation model wins across all 20 tasks, so practitioners should choose models per task rather than by overall score.","Reasoning-reinforced models dominate multi-step conversational QA but cost several times more; for many tasks open-weight mid-scale models offer a better accuracy-per-dollar trade-off.","Numeric labeling and causal reasoning in financial text remain unsolved by zero-shot prompting, pointing to areas where domain-adaptive training is needed.","The public leaderboard and open software make the benchmark a living resource: new models or datasets can be added without rebuilding the pipeline."],"supporting_citations":[{"why":"Supplies the three criteria for holistic evaluation (standardization, multi-metric assessment, recognition of incompleteness) that FLaME adopts as its definition of 'holistic'.","marker":"Liang et al., 2022"},{"why":"The closest prior financial benchmark suite (FinBen) that FLaME compares against in Table 1 and argues falls short of holistic criteria.","marker":"Xie et al., 2024"},{"why":"FLUE, an earlier financial NLP benchmark suite that FLaME lists as lacking standardized pipelines and multi-metric evaluation.","marker":"Shah et al., 2023a"},{"why":"Golden Touchstone, a bilingual financial benchmark compared in Table 1, which FLaME claims lacks recognition of incompleteness and a public leaderboard.","marker":"Wu et al., 2024"},{"why":"FinQA, one of the 20 datasets; supplies the single-turn numerical-reasoning QA task that anchors FLaME's question-answering category.","marker":"Chen et al., 2021"},{"why":"Financial PhraseBank, the sentiment-analysis dataset used as one of the 20 core tasks and a reference point for sentiment results.","marker":"Malo et al., 2013"}],"fun_headline_variants":["23 LMs, 20 finance tasks: FLaME finds no single winner","Open-weight LMs rival big models on finance NLP in FLaME","FLaME benchmark: reasoning-reinforced LMs lead finance Q&A","First holistic finance NLP benchmark FLaME tests 23 LMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparative rankings assume that a single set of zero-shot prompts, tuned on one model and then applied unchanged to 23 models, measures all models fairly; if prompt sensitivity differs across models, the rankings could be an artifact of prompt compatibility rather than of capability.","fun_headline_variants_meta":{"raw":{"variants":["23 LMs, 20 finance tasks: FLaME finds no single winner","Open-weight LMs rival big models on finance NLP in FLaME","FLaME benchmark: reasoning-reinforced LMs lead finance Q&A","First holistic finance NLP benchmark FLaME tests 23 LMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2489,"prompt_tokens":862,"completion_tokens":1627,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":1546}},"tokens_in":478,"tokens_out":1627,"duration_ms":12401,"temperature":1.0,"reasoning_tokens":1546,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:50:04.672755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 20 tasks on a handful of models (for example DeepSeek R1, Claude 3.5 Sonnet, GPT-4o, and Llama 3 8B) with per-model tuned prompts or a few-shot variant, and compare the task-level rankings against the paper's zero-shot rankings; if the top models change materially, the paper's comparative conclusions are prompt-dependent.","supporting_citations":[],"review_version":1}