Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

This paper introduces Fin-RATE, a benchmark showing LLM accuracy drops by 18.6 and 14.35 percentage points as tasks move from single-document reasoning to cross-company comparison and longitudinal tracking, a degradation driven by compariso

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:39 UTC pith:HOHOWK4U

load-bearing objection Solid benchmark with a real design contribution, but the LLM-judge validation is unproven and could bias the headline numbers. the 4 major comments →

arxiv 2602.07294 v4 pith:HOHOWK4U submitted 2026-02-07 cs.CE cs.AI

Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings

Yidong Jiang , Junrong Chen , Eftychia Makri , Jialin Chen , Peiwen Li , Ali Maatouk , Leandros Tassiulas , Eliot Brenner
show 2 more authors
Bing Xiang Rex Ying
This is my paper
classification cs.CE cs.AI
keywords Large Language ModelsFinancial BenchmarkSEC FilingsCross-Entity ReasoningLongitudinal TrackingError TaxonomyRetrieval-Augmented GenerationEvaluation Benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that existing financial benchmarks underestimate LLM capability gaps by treating SEC filings as isolated text spans. To demonstrate this, it builds Fin-RATE, a benchmark of 7,500 questions across three task families that mirror analyst workflows: detail and reasoning within one disclosure, cross-company comparison, and longitudinal tracking of the same firm across years. On 17 LLMs, accuracy falls from roughly 28.8% correct on single-document questions to about 10.1% on cross-company comparison and 14.4% on longitudinal tracking—drops of 18.60 and 14.35 percentage points. A 13-type error taxonomy attributes these drops to comparison hallucination, entity misidentification, temporal mismatches, and query-misunderstanding, and the paper proposes a hierarchical retrieval scheme to improve evidence coverage. If the findings hold, they imply that the main barrier to using LLMs for multi-entity or multi-year financial analysis is not missing facts but the models' inability to align and reason across documents.

Core claim

Fin-RATE is a benchmark constructed from 2,472 SEC filings (10-K, 10-Q, 8-K, and others) covering 43 companies across 36 industries, with 7,500 question-answer pairs split equally across three tasks. The central finding is that LLM accuracy degrades systematically as tasks require aggregation across documents: absolute correct-answer rates drop from an average of 28.76% on single-chunk Detail & Reasoning QA to 10.14% on Enterprise Comparison QA and 14.41% on Longitudinal Tracking QA—the 18.60% and 14.35% drops highlighted in the abstract. Using a three-judge LLM ensemble and a 13-type error taxonomy, the paper shows that these drops are driven by comparative-stance and entity-attribute hallu

What carries the argument

The paper's central object is the Fin-RATE benchmark, a set of three task pipelines—Detail & Reasoning QA (single-chunk), Enterprise Comparison QA (cross-company, same year), and Longitudinal Tracking QA (same company, multiple years)—built from SEC filings and human-validated QA pairs. The other key machinery is a 13-type error taxonomy (four categories: retrieval, generation, finance-specific numeric/semantic, and query/contextual) that attributes each non-correct answer to concrete failure modes such as entity-attribute hallucination, comparative-stance hallucination, and time mismatch. The taxonomy is what lets the paper claim not just that accuracy drops, but why it drops.

Load-bearing premise

The whole measurement pipeline—accuracy drops, error-type frequencies, and the 'retrieval is the bottleneck' conclusion—depends on three LLM judges whose labels were never validated on data they were not also used to tune, and two of those judges are themselves among the models being scored.

What would settle it

Human financial analysts could independently grade a random sample of, say, 1,000 model outputs per task without seeing the LLM judges' labels, then compare the resulting correct-rate drops and error-type counts; if human-graded drops are substantially smaller than 18.60% and 14.35%, or if the error taxonomy looks different, the reported degradation is an artifact of the judge ensemble rather than a genuine capability gap.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Existing single-document benchmarks overstate LLM financial reasoning ability; realistic analyst workflows require cross-entity and cross-year synthesis, where the paper's 17-model evaluation shows large accuracy drops.
  • The 13-type error taxonomy gives a concrete diagnostic: the dominant failure modes are comparison and entity-attribute hallucination (22,885 cases for EC-QA), temporal mismatch and trend hallucination (5.03% and 8.43% for LT-QA), and intent misunderstanding (35.38% of LT-QA errors).
  • Retrieval, not generation, is the main bottleneck in retrieval-augmented settings: end-to-end accuracy stays below 27% under retrieved context while gold-context accuracy reaches 43–57%, and the best retrievers achieve R@10 below 9.73% on EC-QA.
  • Hybrid retrieval that simply fuses BM25 and dense vectors underperforms its components because the two retrievers' candidate sets barely overlap (Jaccard similarity 0.01–0.06); the paper's hierarchical retrieval, which routes queries to company–year buckets, improves recall significantly (e.g., R@10 +12.76 on EC-QA).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A held-out judge validation is the natural next step: two of the three judges are themselves among the models being scored, so re-running the evaluation with a different judge set would test whether the error taxonomy is judge-specific.
  • A concrete extension of the paper's diagnostic value: use the top error types (comparative-stance hallucination, entity misidentification, time mismatch) as negative examples in fine-tuning, then see if Fin-RATE accuracy on EC-QA and LT-QA improves—something the paper does not attempt.
  • The near-zero overlap between sparse and dense retrievers points beyond the paper's hierarchical bucket idea: for any multi-document domain, retrieval should probably be constrained by the document's structure (company, year, section) rather than fused globally, a design principle that could be tested on non-financial corpora.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. Fin-RATE constructs a 7,500-question benchmark from SEC filings, organized into three tasks: Detail & Reasoning QA (single-chunk), Enterprise Comparison QA (cross-company), and Longitudinal Tracking QA (cross-year). The authors describe a 13-type error taxonomy, evaluate 17 LLMs under gold-context and RAG settings, and report headline accuracy drops of 18.60% (DR to EC) and 14.35% (DR to LT), attributing the degradation to comparison hallucinations and entity/temporal mismatches. They also identify retrieval as the dominant bottleneck in RAG and propose a hierarchical retrieval strategy with entity/year bucketing. The dataset and code are promised for release.

Significance. If the evaluation methodology is sound, Fin-RATE fills a real gap: existing SEC-based benchmarks largely test single-document extraction, while Fin-RATE explicitly targets cross-entity and cross-year synthesis, and its error taxonomy goes beyond answer-level correctness. The manuscript is strong in dataset construction detail, human QA review of generated pairs, breadth of evaluated models, and a clear, falsifiable quantitative claim. I verified that the headline 18.60% and 14.35% drops reproduce as unweighted means over Table 2 (DR 28.76% vs EC 10.14%; DR 28.76% vs LT 14.41%). The central risk is that every headline number, error-type count, and Likert-score decline is produced by an LLM-judge ensemble whose weights were tuned and reported on the same sample used for validation, with two of the three judges also being evaluation subjects.

major comments (4)
  1. [§4.2.1, Appendix F, Appendix G] The judge ensemble's fusion weights are grid-searched (step 0.02) on the same 4,500-instance sample used to report the 88.8–90.5% human-agreement numbers, with no held-out validation. Moreover, DeepSeek-V3.2 and Qwen3-235B are themselves rows of Table 2, and GPT-5 receives weight 0.74 on EC-QA and LT-QA while being the best-performing model on those tasks. This creates circularity risk: the 18.60%/14.35% headline drops and all downstream error attributions are computed with a judge ensemble that may be biased toward its own outputs. The authors should report held-out agreement (e.g., cross-validation over the 4,500 instances or a freshly human-labeled sample) and ideally use judges that are not among the evaluated models, or at least measure and report per-judge self-preference effects.
  2. [§4.3.1, Figure 3, Appendix G] The claim that degradation is 'driven by increased comparison hallucinations, temporal and entity mismatches' rests on error-type counts such as 22,885 entity-attribute hallucinations and the 4,834-case increase in comparative-stance hallucination. These counts come from a union rule over the three LLM judges (Appendix G), but Table 14 validates only correctness labels, not error-type labels. If any judge over-tags a subtype, prevalence is inflated. The authors should report per-judge and union-rule error-type agreement against human annotations, ideally on a held-out subset, before using the taxonomy to make causal claims about failure modes.
  3. [§4.3.4, Table 4 vs Table 2] The conclusion that 'the dominant bottleneck in the RAG pipeline is the retriever ... not deficiencies in generation' compares end-to-end accuracy on a 500-example subset (Table 4) with gold-context accuracy on the full 2,500 examples (Table 2), and the 'same models' are not the same across rows: 57.48% is Fin-R1 on DR-QA, while the 43–44% EC/LT figures are GPT-5-websearch. This is an apples-to-oranges comparison. The authors should report gold-context accuracy on the exact 500-example subset used for the RAG evaluation, per model and task, to support the retrieval-bottleneck attribution.
  4. [§4.3.3, Table 3] The sentence 'BM25, which relies on lexical matching, achieves 41.2% recall in DR-QA and 94.3% accuracy in locating year-aligned evidence in LT-QA, outperforming VF at 27.4% and 74.7%' contains a number that is not defined or reported elsewhere. Table 3 reports LT-QA R@10 = 43.86 for BM25 and 20.86 for VF, and no 'year-aligned accuracy' metric appears in the table or text. Please clarify what '94.3%' and '74.7%' measure and provide the corresponding definition and table entry.
minor comments (4)
  1. [Abstract / §4.3.1] The headline 18.60% and 14.35% drops are computed as unweighted means over Table 2 rows, but this aggregation is not stated in the main text. Please make the aggregation explicit, or report per-model drops as well, for reproducibility.
  2. [§3.1 vs Appendix A] Section 3.1 states the corpus covers '43 companies across 36 industries,' but Appendix A and Table 9 describe 34 companies across 7 GICS sectors. Please reconcile these numbers.
  3. [Appendix A.1] The text says the corpus 'totaling 15,311 documents,' but Table 11 sums to 15,311 chunks, and Section 3.1 correctly calls them 'document chunks.' The appendix wording should be corrected.
  4. [§4.2.1 / Appendix G] The ordinal mapping (Failure=0, Incorrect=1, Partial=3, Correct=5) and the threshold rule (score 4–5 maps to Correct, 2–3 to Partial) are reasonable but ad hoc; please justify or evaluate the sensitivity of the headline results to these choices.

Circularity Check

2 steps flagged

Judge reliability is a grid-searched fit on the same 4,500 instances, and the judge panel includes two evaluated models, so the headline drops rest on an unvalidated, partially self-referential measurement.

specific steps
  1. fitted input called prediction [Appendix F / Table 14 / Section 4.2.1]
    "we perform a grid search (with a step size of 0.02) over possible weight combinations to optimize agreement with human annotations. The resulting optimal weights for combining the three Judges are as follows: ... Multi-Judge (Weighted) 88.82 90.41 90.46"

    The reported 88.8–90.5% agreement is the value of the optimization objective on the same 4,500-instance sample used to fit the fusion weights. No held-out validation is described. The paper presents these numbers as 'consistency verification' of the LLM-judge framework, but they are training-set agreements, not independent predictions. Since every Section 4 accuracy number, Likert score, and error-type count is produced by this fused judge ensemble, the reliability of the measurement is a fitted quantity rather than an externally validated one.

  2. other [Section 4.2.1 / Table 2 / Appendix F]
    "Three LLMs with distinct architectures—GPT-5 [26], DeepSeek-V3.2 [8], and Qwen3-235B [32]—serve as independent judges. ... for EC-QA, DeepSeek-V3.2 = 0.140, Qwen3-235B = 0.120, GPT-5 = 0.740"

    The three 'independent judges' are themselves evaluated models in Table 2 (GPT-5-websearch, DeepSeek-V3.2, Qwen3-235B). For EC-QA and LT-QA, GPT-5 receives weight 0.74 in the judge ensemble, and GPT-5-websearch is the highest-scoring model on those tasks (43.64% and 43.52%). Thus the measured performance of the top model is substantially determined by that same model's own judgments. The label 'independent' is contradicted by the paper's own tables, creating a self-referential loop in the evaluation of the central capability claim.

full rationale

The benchmark construction itself is largely independent: the QA pairs are grounded in SEC filings, filtered by regex/LLM checks, and manually validated with explicit criteria (Appendix E), and the gold answers and key points are external to the evaluated models. The headline 18.60% and 14.35% drops are simple arithmetic over Table 2 and are internally reproducible. The circularity is confined to the evaluation layer. First, the judge-fusion weights are grid-searched on the same 4,500 instances used to report human agreement, so the '88.8–90.5%' figure is a fitted optimum, not a held-out accuracy. Second, the judge panel overlaps with the evaluated model set, and GPT-5 dominates the grading on the exact tasks where it is reported best. These issues do not make the dataset derivation circular, but they mean the central 'performance degradation / error taxonomy' conclusions rest on a measurement instrument whose reliability is self-fit and whose independence is compromised. No load-bearing self-citation chain or uniqueness-theorem argument appears in the paper. The core benchmark content has independent value; the circularity score reflects the unvalidated and partially self-referential judge pipeline rather than the dataset construction.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 3 invented entities

The paper's numbers rest on three LLM-mediated instruments: LLM-generated QA pairs verified by LLMs and humans, LLM judges with weights fit on the validation sample, and an error taxonomy derived from the same output distribution it is used to count. None of these are parameter-free; the judge weights are explicit fitted parameters (Appendix F). The SEC-item alignment and judge-approximates-human assumptions are domain assumptions that a re-implementation would have to accept or re-test.

free parameters (3)
  • Judge fusion weights (per task) = DR: {DeepSeek-V3.2: 0.70, Qwen3-235B: 0.10, GPT-5: 0.20}; EC/LT: {DeepSeek-V3.2: 0.14, Qwen3-235B: 0.12, GPT-5: 0.74}
    Grid-searched (step 0.02) on a 4,500-instance sample to maximize agreement with human labels, then used for all 7,500×17 outputs; in-sample fit with no held-out validation (Appendix F).
  • Ordinal label mapping for judge fusion = Failure=0, Incorrect=1, Partial=3, Correct=5; fused 2–3→Partial, 4–5→Correct
    Hand-chosen mapping and thresholds in Appendix G; the 2–3/4–5 boundaries directly determine the final accuracy numbers.
  • RAG context size and subset filter = top-5 chunks; EC/LT restricted to questions with ≤5 gold chunks
    Hand-chosen in §4.3.4; biases the RAG subset toward easier questions, which makes the retrieval-bottleneck conclusion conservative rather than inflated.
axioms (5)
  • domain assumption SEC structured headings (e.g., Item 1A, Item 7) are semantically aligned across companies so that summary-based comparison QA is well-defined
    Invoked in §3.2.2 EC-QA construction ('pinpoint semantically aligned sections using structured headings'); if disclosure styles are too heterogeneous, gold comparisons rest on an LLM's forced alignment.
  • domain assumption A weighted panel of three LLM judges (weights tuned in-sample) approximates human expert judgment for correctness, Likert, and error-type labels
    Invoked in §4.2.1 and Appendix F; all quantitative results depend on this.
  • domain assumption LLM-generated QA pairs (DeepSeek-V3.2) filtered by LLMs and reviewed by humans yield unbiased, answerable ground truth
    Invoked in §3.3; with ~7,500 pairs and no inter-annotator agreement reported, LLM-mediated question design could imprint systematic biases (e.g., favoring comparison-friendly phrasing).
  • domain assumption EDGAR-to-Markdown conversion preserves all content needed for QA
    Invoked in §3.1; tables and lists are converted, but no fidelity audit is reported.
  • domain assumption Gold chunk sets for EC/LT (from generation-time selection) are complete for retrieval evaluation
    Invoked in §4.2.2 metrics; if gold chunk sets are under-specified, low Recall/Precision (e.g., R@10 ≤ 9.73% for EC-QA) partly reflects annotation incompleteness, weakening the 'retrieval is the bottleneck' conclusion.
invented entities (3)
  • Fin-RATE dataset (7,500 QA pairs; ~15,311 chunks from ~2,472 filings) independent evidence
    purpose: New evaluation instrument measuring single-document, cross-entity, and longitudinal financial reasoning
    Released on HuggingFace/GitHub; externally checkable QA pairs and corpus, so claims can be falsified by re-testing.
  • 13-type error taxonomy (B1–B4, C1–C4, D1–D2 with subtypes) no independent evidence
    purpose: Diagnostic categories for attributing LLM failures to retrieval, generation, numerics, or comprehension
    Derived from manual examination of outputs from the same model distribution it is later applied to by LLM judges; no independent human-labeled error corpus is released for validating the taxonomy itself.
  • Hierarchical retrieval (entity/year bucket routing) independent evidence
    purpose: Improve EC/LT evidence coverage via structure-aware search
    Falsifiable via standard IR metrics (R@10, P@1, MRR) on the released dataset; improvements are benchmarked against BM25/reranker baselines.

pith-pipeline@v1.3.0-alltime-deepseek · 48502 in / 25451 out tokens · 233018 ms · 2026-08-03T03:39:27.286110+00:00 · methodology

0 comments
read the original abstract

With the increasing deployment of Large Language Models (LLMs) in the finance domain, LLMs are increasingly expected to parse complex regulatory disclosures. However, existing benchmarks often focus on isolated details, failing to reflect the complexity of professional analysis that requires synthesizing information across multiple documents, reporting periods, and corporate entities. Furthermore, these benchmarks do not disentangle whether errors arise from retrieval failures, generation inaccuracies, domain-specific reasoning mistakes, or misinterpretation of the query or context, making it difficult to precisely diagnose performance bottlenecks. To bridge these gaps, we introduce Fin-RATE, a benchmark built on U.S. Securities and Exchange Commission (SEC) filings and mirroring financial analyst workflows through three pathways: detail-oriented reasoning within individual disclosures, cross-entity comparison under shared topics, and longitudinal tracking of the same firm across reporting periods. We benchmark 17 leading LLMs, spanning open-source, closed-source, and finance-specialized models, under both ground-truth context and retrieval-augmented settings. Results show substantial performance degradation, with accuracy dropping by 18.60% and 14.35% as tasks shift from single-document reasoning to longitudinal and cross-entity analysis. This degradation is associated with increased comparison hallucinations, temporal and entity mismatches, and is further reflected in declines in reasoning quality and factual consistency--limitations that existing benchmarks have yet to formally categorize or quantify.

Figures

Figures reproduced from arXiv: 2602.07294 by Ali Maatouk, Bing Xiang, Eftychia Makri, Eliot Brenner, Jialin Chen, Junrong Chen, Leandros Tassiulas, Peiwen Li, Rex Ying, Yidong Jiang.

Figure 1
Figure 1. Figure 1: An overview of Fin-RATE with core tasks and cor [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the Fin-RATE dataset construction framework. Stage (1) builds a heterogeneous corpus from SEC filings [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of QA error subtypes across three task Left: Level 1 error types, covering broad categories B1–B4, C1–C4, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Radar charts showing model categories’ perfor [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Retrieval error distribution. ME = Missing Evidence, [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance comparison (R@10, P@1, MRR) be [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Pie chart illustrating the distribution of the 34 sam [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Pie chart illustrating the state-wise distribution of [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Level 1 and Level 2 error distributions for Closed-Source models [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Level 1 and Level 2 error distributions for Open-Source General models [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Level 1 and Level 2 error distributions for Open-Source Finance models [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Example for Detail & Reasoning QA (DR) question [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Example for Enterprise Comparison (EC) question [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Example for Longitudinal Tracking (LT) question [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MacroLens: A Multi-Task Benchmark for Contextual Financial Reasoning under Macroeconomic Scenarios

    cs.LG 2026-06 unverdicted novelty 7.0

    MacroLens is a point-in-time multi-signal benchmark dataset and seven tasks for evaluating contextual financial reasoning models under macroeconomic scenarios.

  2. TRACE: Tourism Recommendation with Accountable Citation Evidence

    cs.IR 2026-05 unverdicted novelty 7.0

    TRACE is a new benchmark dataset and evaluation suite for conversational tourism recommenders that requires systems to suggest POIs, cite verifiable review spans, and recover from rejections, revealing a Three-Compete...

  3. FinSAgent: Corpus-Aligned Multi-Agent RAG Framework for Evidence-Grounded SEC Filing Question Answering

    cs.IR 2026-07 conditional novelty 6.0

    FinSAgent improves financial filing QA by conditioning sub-queries on a summary of the local corpus and gating semantic reranking with a learned validity signal, beating baseline systems on five benchmarks.

Reference graph

Works this paper leans on

123 extracted references · 2 canonical work pages · cited by 3 Pith papers

  1. [1]

    Noga BenYoash, Menachem Brief, Oded Ovadia, Gil Shenderovitz, Moshik Mishaeli, Rachel Lemberg, and Eitam Sheetrit. 2025. SECQUE: A Benchmark for Evaluating Real-World Financial Analysis Capabilities. InProceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM2), Ofir Ar- viv, Miruna Clinciu, Kaustubh Dhole, Rotem Dror, Sebastian Gehrma...

    2025

  2. [2]

    CeADAR. 2023. FinanceConnect-13B (Revision 5f7841d). doi:10.57967/hf/1405

    doi:10.57967/hf/1405 2023

  3. [3]

    Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu

  4. [4]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A Dataset of Numerical Reasoning over Financial Data. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang...

    2021

  5. [5]

    Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. ConvFinQA: Exploring the Chain of Numerical Rea- soning in Conversational Finance Question Answering. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association f...

    doi:10.18653/v1/2022.emnlp- 2022

  6. [6]

    2025.FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation

    Chanyeol Choi, Jihoon Kwon, Jaeseon Ha, Hojun Choi, Chaewoon Kim, Yongjae Lee, Jy-yong Sohn, and Alejandro Lopez-Lira. 2025.FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation. Association for Computing Machinery, New York, NY, USA, 638–646. https://doi.org/10. 1145/3768292.3770361

    arXiv 2025

  7. [7]

    2025.How finance teams are putting AI to work to- day

    McKinsey & Company. 2025.How finance teams are putting AI to work to- day. https://www.mckinsey.com/capabilities/strategy-and-corporate-finance/ our-insights/how-finance-teams-are-putting-ai-to-work-today Accessed: 2026- 02-08

    2025

  8. [8]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

    Pith/arXiv arXiv 2025

  9. [9]

    Yang Deng, Wenqiang Lei, Wenxuan Zhang, Wai Lam, and Tat-Seng Chua. 2022. PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Ling...

    doi:10.18653/v1/2022.emnlp-main.469 2022

  10. [10]

    Elastic. [n. d.]. elasticsearch (GitHub repository). https://github.com/elastic/ elasticsearch. https://github.com/elastic/elasticsearch Accessed: 2026-01-20

    2026

  11. [11]

    FinLang. 2024. Finance embeddings investopedia. https://huggingface.co/ FinLang/finance-embeddings-investopedia. https://huggingface.co/FinLang/ finance-embeddings-investopedia Accessed: 2026-01-20

    2024

  12. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava...

    Pith/arXiv arXiv 2026

  13. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, ...

    2025

  14. [14]

    Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, Zhiqiang Liu, Sizhe Wang, Jinyi Niu, Chuqi Wang, Yanhui Wang, Xiaolong Liang, Xiaoming Huang, Bing Zhu, Zhongyu Wei, Yun Chen, Weining Shen, and Liwen Zhang. 2025. FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models. InProceedings of the 2025 Conference of t...

    2025

  15. [15]

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. FinanceBench: A New Benchmark for Financial Question Answering. arXiv:2311.11944 [cs.CL]

    Pith/arXiv arXiv 2023

  16. [16]

    Lanlan Ji, Dominic Seyler, Gunkirat Kaur, Manjunath Hegde, Koustuv Das- gupta, and Bing Xiang. 2025. PHANTOM: A Benchmark for Hallucination Detection in Financial Long-Context QA. InThe Thirty-ninth Annual Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=5YQAo0S3Hm

    2025

  17. [17]

    Viet Lai, Michael Krumdick, Charles Lovering, Varshini Reddy, Craig Schmidt, and Chris Tanner. 2025. SEC-QA: A Systematic Evaluation Corpus for Financial QA. In Proceedings of The 10th Workshop on Financial Technology and Natural Language Processing, Chung-Chi Chen, Genta Indra Winata, Stephen Rawls, Anirban Das, Hsin-Hsi Chen, and Hiroya Takamura (Eds.)....

    doi:10.18653/v1/2025.finnlp-2.15 2025

  18. [18]

    Chuang Liu, Junzhuo Li, and Deyi Xiong. 2023. Tab-CQA: A Tabular Conver- sational Question Answering Dataset on Financial Reports. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), Sunayana Sitaram, Beata Beigman Klebanov, and Jason D Williams (Eds.). Association for Computational Linguis...

    work page doi:10.18653/v1/2023.acl-industry.20 2023

  19. [19]

    Zhaowei Liu, Xin Guo, Zhi Yang, Fangqi Lou, Lingfeng Zeng, Mengping Li, Qi Qi, Zhiqiang Liu, Yiyang Han, Dongpo Cheng, Ronghao Chen, Huacan Wang, Xingdong Feng, Huixia Judy Wang, Chengchun Shi, and Liwen Zhang. 2026. Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning. arXiv:2503.16252 [cs.CL] https://arxiv.org/abs/2503.16252

    arXiv 2026

  20. [20]

    Zheng Liu, Chaofan Li, Shitao Xiao, Yingxia Shao, and Defu Lian. 2025. Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval. arXiv:2312.15503 [cs.CL] https://arxiv.org/abs/2312.15503

    arXiv 2025

  21. [21]

    OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Apple- baum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives,...

    Pith/arXiv arXiv 2025

  22. [22]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Bal- com, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Bern...

    Pith/arXiv arXiv 2026

  23. [23]

    Lingfei Qian, Weipeng Zhou, Yan Wang, Xueqing Peng, Han Yi, Yilun Zhao, Jimin Huang, Qianqian Xie, and Jian yun Nie. 2025. Fino1: On the Transfer- ability of Reasoning-Enhanced LLMs and Reinforcement Learning to Finance. arXiv:2502.08127 [cs.CL] https://arxiv.org/abs/2502.08127

    Pith/arXiv arXiv 2025

  24. [24]

    Varshini Reddy, Rik Koncel-Kedziorski, Viet Dac Lai, Michael Krumdick, Charles Lovering, and Chris Tanner. 2024. DocFinQA: A Long-Context Financial Rea- soning Dataset. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Co...

    doi:10.18653/v1/2024.acl-short.42 2024

  25. [25]

    Stephen Robertson, Hugo Zaragoza, et al . 2009. The probabilistic relevance framework: BM25 and beyond.Foundations and trends®in information retrieval 3, 4 (2009), 333–389

    2009

  26. [26]

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al

  27. [27]

    2025.Can AI Read Like a Financial Analyst? A Financial Touchstone for Frontier Language Models Such as Gemini 2.5 Pro, o3, and Grok 4 on Long-Context Annual Report Comprehension

    Jan Spörer. 2025.Can AI Read Like a Financial Analyst? A Financial Touchstone for Frontier Language Models Such as Gemini 2.5 Pro, o3, and Grok 4 on Long-Context Annual Report Comprehension. Association for Computing Machinery, New York, NY, USA, 291–298. https://doi.org/10.1145/3768292.3770417

    arXiv 2025

  28. [28]

    Zichen Tang, E Haihong, Ziyan Ma, Haoyang He, Jiacheng Liu, Zhongjun Yang, Zihua Rong, Rongjin Li, Kun Ji, Qing Huang, et al . 2025. Financereasoning: Benchmarking financial numerical reasoning more credible, comprehensive and challenging. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 15...

    2025

  29. [29]

    Core Team, Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, Gang Xie, Hailin Zhang, Hanglong Lv, Hanyu Li, Heyu Chen, Hongshen Xu, Houbin Zhang, Huaqiu Liu, Jiangshan Duo, Jianyu Wei, Jiebao Xiao, Jinhao Dong, Jun Shi, Junhao Hu, Kainan Bao, Kang Zhou, Lei Li, Liang Zhao, Linghao Zhang,...

    Pith/arXiv arXiv 2026

  30. [30]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre- Trained Transformers. arXiv:2002.10957 [cs.CL] https://arxiv.org/abs/2002.10957

    Pith/arXiv arXiv 2020

  31. [31]

    Xiaojun Wu, Junxi Liu, Huan-Yi Su, Zhouchi Lin, Yiyan Qi, Chengjin Xu, Jiajun Su, Jiajie Zhong, Fuwei Wang, Saizhuo Wang, Fengrui Hua, Jia Li, and Jian Guo

  32. [32]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv 2025

  33. [33]

    Mengao Zhang, Jiayu Fu, Tanya Warrier, Yuwen Wang, Tianhui Tan, and Ke-wei Huang. 2025. FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance. InProceedings of the 6th ACM International Conference on AI in Finance. 159–167

    2025

  34. [34]

    InFindings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.)

    Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 22544–22560. doi:10.18653/v1/2025.fi...

    work page doi:10.18653/v1/2025.findings-emnlp.1227 2025

  35. [37]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A Question An- swering Benchmark on a Hybrid of Tabular and Textual Content in Finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natur...

    2021

  36. [39]

    MUST explicitly mention both company names: {company1} and {company2}

  37. [40]

    MUST explicitly mention the year: {year}

  38. [41]

    MUST focus on comparative analysis between the two companies

  39. [42]

    MUST be specific and directly related to the provided context information

  40. [43]

    MUST strictly follow the QUESTION TYPE instruction above𝑠𝑖𝑛𝑔𝑙𝑒𝑖𝑡𝑒𝑚𝑑𝑒𝑝𝑡ℎ𝑣𝑠.𝑚𝑢𝑙𝑡𝑖−𝑖𝑡𝑒𝑚𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑖𝑜𝑛

  41. [44]

    MUST align with the focus guidance provided above

  42. [45]

    MUST be suitable for financial or business analysis

  43. [47]

    MUST address different aspects to ensure variety across the {num_pairs} questions

  44. [48]

    For multi-item combinations: Each question MUST ref- erence multiple sections and show their interrelationship

  45. [49]

    CRITICAL: MUST only ask about topics where BOTH companies have information - verify this before generating the question REQUIREMENTS FOR EACH ANSWER:

  46. [50]

    MUST be comprehensive and detailed

  47. [51]

    MUST be based on the provided context information

  48. [52]

    MUST compare aspects between {company1} and {company2}

  49. [53]

    MUST be written in clear, professional English

  50. [54]

    MUST provide specific insights and comparisons

  51. [55]

    For multi-item combinations: Answers MUST integrate information from multiple sections

  52. [56]

    not mentioned

    CRITICAL: MUST provide information for BOTH companies - DO NOT use phrases like "not mentioned" , "not available" , "not disclosed" , "information is not provided" , or similar statements indicating missing information

  53. [57]

    qa_pairs

    If you cannot provide information for both companies, DO NOT generate this question-answer pair - skip it and generate a different one IMPORTANT:Ensure questions are DISTINCT and cover different aspects. For single items, vary the specific de- tails being explored. For combinations, vary the types of relationships being analyzed. Generate exactly {num_pai...

    2026

  54. [58]

    Drill deep into specific risk categories (operational, fi- nancial, regulatory, market, etc.)

  55. [59]

    Compare detailed risk descriptions, quantification meth- ods, and disclosure specificity

  56. [60]

    Analyze granular differences in risk prioritization and presentation styles

  57. [61]

    Focus on concrete examples and specific risk factors mentioned in each company’s disclosure

  58. [62]

    Avoid general comparisons - instead ask about specific risk types, mitigation strategies, or risk assessment method- ologies

  59. [63]

    Questions should be narrow and detailed, exploring depth within this single section <|eot_id|><|start_header_id|>assistant <|end_header_id|> Item 7. Management’s Discussion and Analysis of Financial Condition and Results of Operations - In- dividual Item Focus <|begin_of_text|><|start_header_id|>system <|end_header_id|> INDIVIDUAL ITEM FOCUS - Generate qu...

  60. [64]

    Focus on specific financial metrics, trends, or perfor- mance indicators discussed in MD&A

  61. [65]

    Compare detailed explanations of revenue drivers, cost factors, or operational changes

  62. [66]

    Analyze specific forward-looking statements, guidance, or management outlook

  63. [67]

    Explore granular differences in how each company ex- plains specific financial results or operational events

  64. [68]

    Ask about concrete examples, specific periods, or partic- ular business segments discussed

  65. [69]

    Legal Proceedings - Individual Item Focus <|begin_of_text|><|start_header_id|>system <|end_header_id|> INDIVIDUAL ITEM FOCUS - Generate questions that:

    Questions should be narrow and detailed, exploring depth within this single section <|eot_id|><|start_header_id|>assistant <|end_header_id|> Item 3. Legal Proceedings - Individual Item Focus <|begin_of_text|><|start_header_id|>system <|end_header_id|> INDIVIDUAL ITEM FOCUS - Generate questions that:

  66. [70]

    Focus on specific legal cases, claims, or proceedings men- tioned by each company

  67. [71]

    Compare detailed information about case types, poten- tial outcomes, or settlement amounts

  68. [72]

    Analyze specific litigation strategies, legal reserves, or disclosure practices Under Review, February, 2026 Yidong Jiang et al

    2026

  69. [73]

    Explore granular differences in how each company cate- gorizes or prioritizes legal matters

  70. [74]

    Ask about concrete examples of specific legal proceed- ings or their potential impacts

  71. [75]

    Questions should be narrow and detailed, exploring depth within this single section <|eot_id|><|start_header_id|>assistant <|end_header_id|> Item 10. Directors, Executive Officers, and Corporate Governance - Individual Item Focus <|begin_of_text|><|start_header_id|>system <|end_header_id|> INDIVIDUAL ITEM FOCUS - Generate questions that:

  72. [76]

    Focus on specific governance structures, board commit- tees, or executive roles

  73. [77]

    Compare detailed information about board composition, director qualifications, or executive compensation

  74. [78]

    Analyze specific governance policies, codes of conduct, or corporate governance practices

  75. [79]

    Explore granular differences in how each company struc- tures its governance framework

  76. [80]

    Ask about concrete examples of governance mechanisms or specific governance-related disclosures

  77. [81]

    Questions should be narrow and detailed, exploring depth within this single section <|eot_id|><|start_header_id|>assistant <|end_header_id|> Multi-item Focus The multi-item focus configuration enablesintegrated, rela- tional analysisacross multiple annual report sections by explicitly connecting information from different areas to generates questions that...

  78. [82]

    EXPLICITLY connect information from BOTH Item 1 (Business) AND Item 8 (Financial Statements)

  79. [83]

    Ask about HOW business operations (from Item 1) trans- late into specific financial metrics (from Item 8)

  80. [84]

    Compare the relationship between business model char- acteristics and financial performance indicators

Showing first 80 references.