Pith. sign in

REVIEW 3 major objections 5 minor 16 references

FORCE-Bench, a 251-query, rubric-scored benchmark for agentic AI in enterprise finance, claims that under operational latency constraints a purpose-built finance agent is more reliable across quality dimensions than general-purpose agentic

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:22 UTC pith:7F2O2AZJ

load-bearing objection A worthwhile enterprise-finance agent benchmark, but the headline Finance-Agent-vs-generalist comparison is so contaminated by co-development that the paper should reframe around the benchmark artifact. the 3 major comments →

arxiv 2607.19409 v1 pith:7F2O2AZJ submitted 2026-07-11 cs.AI

FORCE-Bench: A Benchmark, Dataset, and Evaluation Harness for Agentic AI in Enterprise Finance

Wolfgang M. Pauli , Sarah Panda , Kidus Admassu , Said Bleik , Ademola Okerinde , Jeremy Reynolds This is my paper
classification cs.AI
keywords agentic AIenterprise financebenchmarkERPrubric-based evaluationLLM judgefinancial obligation researchbusiness brief generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces FORCE-Bench, a benchmark for agentic AI in enterprise finance containing 251 expert-annotated queries across three task types: querying ERP systems for receivables and payables, answering time-bound questions from public filings and market data, and generating multi-source business briefs. Instead of a binary pass/fail, each response is scored along eight rubric dimensions—accuracy, citations, clarity, depth, groundedness, recency, relevance, and structure—using an LLM judge. Under a 60-second time budget per query (300 seconds for business briefs), general-purpose agentic systems do not consistently meet finance-domain quality standards, while the purpose-built finance agent is more reliable across dimensions. The dataset, rubrics, harness, and analysis code are released for open use.

Core claim

The central claim is that reliable AI in enterprise finance requires workflow-oriented, rubric-based evaluation, not just general capability or task-success metrics. FORCE-Bench operationalizes this by pairing 251 real-world finance queries with assertion-level rubrics that reflect what finance professionals care about—for example, that the response cites a data source by name or grounds financial figures in cited sources. Under common tool access and a tight latency budget, the paper finds a purpose-built finance agent outperforms general-purpose agentic runtimes on all eight dimensions, with the largest margin on ERP accuracy (0.77 versus 0.40–0.67 for general-purpose systems). The paper a

What carries the argument

The key machinery is the rubric-based evaluation framework. Each query carries a set of assertions—testable statements about what a finance professional would expect in a response—and an LLM judge scores each assertion on a continuous 0–1 scale; a dimension score is the mean of its assertions. This converts messy agent outputs into diagnostic, comparable scores across eight dimensions, and it is decoupled from any particular agent so practitioners can apply it to their own datasets. A linear mixed model with per-question random intercepts then compares providers while accounting for question difficulty.

Load-bearing premise

The load-bearing premise is that the single LLM judge used for scoring produces unbiased, faithful quality scores across all providers; if this judge favors outputs similar to its own family, the reported cross-system margins—and possibly rankings—could be distorted.

What would settle it

Have independent finance professionals manually score the same 251 responses on the same rubric; if the purpose-built agent's lead over general-purpose systems does not replicate in human ratings, then the LLM-judge-based comparison does not support the reliability claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Finance teams can adopt FORCE-Bench to compare agents on the specific workflows (ERP obligations, entity research, brief synthesis) they plan to deploy, rather than relying on general-purpose benchmarks.
  • Latency budgets are a first-class hyper-parameter: a 60-second ceiling materially caps quality for multi-step retrieval, so deployment decisions should jointly consider reasoning effort and timeout.
  • Groundedness is a highly discriminating dimension, with general-purpose systems scoring near zero when their retrieval pipelines do not capture source content; improving source-attribution pipelines is a concrete route to better performance.
  • Because the harness is decoupled from any agent implementation, the rubrics can be adapted to other enterprise domains beyond finance (for example, legal or supply chain) with comparable workflows.
  • The benchmark's per-dimension scores reveal asymmetric strengths—all systems score near-perfect citations on business briefs but differ sharply on depth—so improvement efforts can be targeted precisely.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If judge self-preference bias exists and is unmeasured, the absolute margins could shrink, but the paper's own cross-judge analysis on one run suggests relative rankings are robust; a systematic multi-judge audit across all providers would settle this.
  • The paper only measures accuracy on ERP-grounded tasks; an inference is that for public-data tasks, groundedness serves as a proxy for factual correctness, and one could develop automated fact-checking against contemporaneous filings to extend accuracy measurement.
  • The interaction between reasoning effort and timeout suggests that for latency-sensitive deployment, deliberately throttling reasoning effort may be a quality-preserving choice—a hypothesis practitioners could test on their own workloads.
  • A multi-turn extension would be a natural next step, since real finance sessions involve clarification and context carryover; a dataset of follow-up queries on top of these 251 would test memory and clarification behavior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. FORCE-Bench is a benchmark and evaluation harness for agentic AI in enterprise finance, comprising 251 expert-annotated queries across three task types: financial obligation research over a synthetic Dynamics 365 ERP, financial entity performance research from public filings/market data, and business brief generation. Responses are scored by an LLM judge (GPT-5.2) on a rubric with up to eight dimensions (accuracy, citations, clarity, depth, groundedness, recency, relevance, structure). The paper compares a purpose-built Finance Agent for Microsoft 365 Copilot with two general-purpose harnesses (Anthropic Claude CLI and OpenAI Responses API) under shared tool access and latency budgets. The headline claim is that the Finance Agent is 'more reliable across dimensions' than the general-purpose systems, with statistical comparisons via linear mixed models and a timeout ablation.

Significance. The benchmark resource itself is valuable: it addresses an under-served area, provides expert-annotated queries, deterministic ERP ground truth for the obligation task, open-source dataset/rubrics/harness, and a sensible LMM-based significance framework. The authors are also unusually transparent about limitations, including benchmark–agent co-development, judge bias, and single-turn scope. However, the central comparative claim is currently not supported as stated: the Finance Agent was iteratively developed on the same benchmark and rubric signals used in the final evaluation, while the general-purpose comparators were not. This is a direct confound for the abstract's unqualified claim. The LLM-judge issue is secondary but also incompletely resolved. With reframing or additional validation, the contribution could be acceptable; as written, the paper requires major revision.

major comments (3)
  1. [§5, §3.1, §3.3, Abstract] The paper discloses (Section 5) that the Finance Agent 'was developed iteratively alongside the benchmark' and (Section 3.1) that the dataset was 'continuously updated based on evaluation results,' while the rubric was 'calibrated against real model outputs' (Section 3.3). The general-purpose comparators receive only the shared instructions and tools (Section 3.4), so Table 1 and the abstract's 'more reliable across dimensions' claim compare a system with direct exposure to the final queries/rubric against systems without such exposure. This is a train/test leakage; the Section 5 concession that the advantage is 'partly a product of this co-development' does not remove the problem because the abstract and Section 4.1 state the lead unconditionally. To make the comparison load-bearing, the authors should either freeze a benchmark snapshot before further agent development and report a held
  2. [§3.2, §5, Appendix A] Section 3.2 fixes OpenAI GPT-5.2 as the judge for all runs, and one of the compared systems is OpenAI GPT-5.5. Appendix A validates rank agreement on a single Claude Haiku run only and explicitly cannot rule out self-preference bias. The ERP accuracy cell (ρ=0.944) is reassuring, but the headline spans eight dimensions and four systems; absolute margins in Table 1 (e.g., Groundedness 0.76 vs 0.33–0.53) are all produced by this one judge. Please add cross-judge scoring (at least a second non-OpenAI judge, ideally human spot-checks) for all systems and dimensions, or provide bias-corrected confidence intervals; otherwise the quantitative margins are not fully interpretable.
  3. [§4.2.3, Figure 3, Table 1] Section 4.2.3 and Figure 3 report near-zero groundedness for general-purpose systems on finance_qa and business_brief, with the caption noting this is due to absence of captured source content from their retrieval pipelines, 'not necessarily a reflection of factual error.' However, Section 3.2 designates groundedness as the primary factual-grounding signal for these task types. Scoring an unmeasurable quantity as zero and then aggregating it into the Overall row (Table 1) conflates instrumentation failure with quality failure. Please either instrument the general-purpose harnesses so their tool/source traces are captured like the Finance Agent's, or mark groundedness as 'not measurable' for those harnesses and recompute overall scores without it.
minor comments (5)
  1. [§3.4 vs Figure 1] The Figure 1 caption says Claude systems were run with 'reasoning effort = low,' but Section 3.4 says the Claude CLI was invoked 'without an explicit effort parameter, corresponding to standard mode.' Please align these descriptions and define whether 'standard' equals 'low.'
  2. [§3.2, Table 1] The 'eight dimensions' are a union over task types rather than a set measured everywhere. Please provide a clear dimension-by-task applicability table and state explicitly how missing dimensions enter the Overall score.
  3. [Appendix A] Report the number of matched pairs per (task-type, metric) cell. With n=25 in business brief cells, Spearman correlations are unstable and undefined at ceiling; the current '25–150' range is too coarse for interpreting the low-ρ cells.
  4. [Appendix B, Figure 9] The possibility that the Claude CLI did not fully capture tool calls for Opus on finance_qa (median 0 calls) should be mentioned in the main results, since it directly affects interpretation of groundedness and citations for that provider.
  5. [Table 1, §3.4.1] The aggregation formula for the 'Overall' row is not specified. Please give the exact weighting across dimensions and task types (e.g., unweighted mean of cell means, or task-type-equal then dimension-mean), so readers can reproduce Table 1 from the per-cell scores.

Circularity Check

1 steps flagged

The Finance Agent's reported superiority is partly self-confirming: the agent and benchmark were co-developed, making the central comparison partially in-sample.

specific steps
  1. fitted input called prediction [Section 5 (Limitations) and Section 3.1 (Dataset Curation)]
    "The Finance Agent was developed iteratively alongside the benchmark: dataset coverage was expanded based on evaluation results, and SME feedback on model outputs directly informed rubric refinement (Section 3.1). The Finance Agent therefore benefited from direct exposure to benchmark signals that the general-purpose comparators did not receive, and its performance advantage should be understood partly as a product of this co-development rather than solely as an architectural difference."

    The benchmark is both the instrument and the development signal: Section 3.1 says the dataset was continuously updated based on evaluation results, and Section 5 concedes the Finance Agent had direct exposure to the benchmark queries and rubrics that the general-purpose comparators did not receive. The headline comparison is therefore not a blind, out-of-sample test. The agent's design, prompts, and tool routing were adapted to the exact evaluation set; reporting its higher scores as evidence of general reliability re-imports the development objective as a result. The gap may be partly real, but it is an in-sample fit rather than an independent prediction, so the claimed comparative advantage is partially circular by construction.

full rationale

FORCE-Bench is a benchmark paper rather than a formal derivation, so most classic circularity patterns—equation-level reductions, imported uniqueness theorems, ansatz smuggling, and renaming known results—do not apply. No load-bearing self-citation chain is present. The main circularity concern is the explicitly acknowledged co-development of the Finance Agent and the benchmark: Section 5 states the agent was iteratively modified using the same queries and rubric signals used in the final evaluation, and Section 3.1 confirms the dataset was continuously updated based on evaluation results. This creates a partial fitted-input-called-prediction problem: the central comparative claim is not an independent out-of-sample measurement but partly a restatement of the development objective. The LLM-judge issue is a separate validity threat, not a definitional circularity; the appendix's rank-agreement check and explicit disclosure prevent it from being a by-construction reduction. The score of 5 reflects one substantial partial-circularity step without full by-construction equivalence.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

No new physical or theoretical entities are postulated; the eight evaluation dimensions are measurement constructs, not invented entities. The load-bearing assumptions are the judge's impartiality, the rubric's construct validity, ERP data fidelity, LMM inferential assumptions, and the representativeness of the chosen operational constraints.

free parameters (2)
  • timeout budget = 60 s for obligation/entity research; 300 s for business briefs
    Hand-chosen operational constraint (Section 3.4); ablation in Section 4.3 shows the timeout materially changes comparative scores, so this hand-set parameter shapes the headline result.
  • reasoning effort for general-purpose harnesses = low for OpenAI Responses API; standard (no extended thinking) for Claude CLI
    Hand-set configuration that caps general-purpose systems (Section 3.4); Section 4.3 ablation shows relaxing effort narrows the gap with the Finance Agent, so this choice materially influences the comparison.
axioms (5)
  • domain assumption GPT-5.2 judge scores are a faithful, near-unbiased measurement of finance-domain quality across providers.
    All headline comparisons use this single judge (Section 3.2); Appendix A checks only one inference run and explicitly cannot rule out self-preference bias.
  • domain assumption The eight rubric dimensions and their assertions capture what finance professionals need.
    Rubric was built from SME input and survey feedback and then calibrated against model outputs (Section 3.3), so its validity rests on that process being representative.
  • domain assumption The synthetic Dynamics 365 Finance data is structurally faithful to real enterprise ERP environments.
    ERP accuracy claims generalize only if the synthetic 1,000-customer/1,000-vendor environment (Section 3.3.1) is representative of real deployments.
  • standard math Linear mixed model asymptotic assumptions support the significance tests.
    Section 3.4.1 uses ML estimation and Wald contrasts; validity assumes asymptotic normality, correct random-effects structure, and ignorable listwise missingness.
  • domain assumption The chosen deployment constraints (60s/300s timeouts, low reasoning effort) represent real operational conditions.
    The central claim 'under operational constraints' depends on these settings; if real deployments allow longer latency, the ordering changes (Section 4.3).

pith-pipeline@v1.3.0-alltime-deepseek · 16527 in / 12882 out tokens · 139937 ms · 2026-08-02T07:22:41.217248+00:00 · methodology

0 comments
read the original abstract

Recent advances in large language models have accelerated deployment of agentic systems in operational finance. Existing benchmarks emphasize measuring general capabilities, instruction following, or safety, but few directly address the operational finance workflows that agentic systems are now being deployed to automate. Finance professionals require agents to not only provide factually sound and properly grounded information, but also ensure that this information is verifiable and consistently adheres to rules and constraints of the operational finance domain. We introduce FORCE-Bench, which contains 251 expert-annotated queries and evaluates responses using a rubric-based framework calibrated to the requirements of the operational finance domain, across eight dimensions: accuracy, citations, clarity, depth, groundedness, recency, relevance, and structure. FORCE-Bench assesses agentic systems on three task types: financial obligation research (querying ERP systems for accounts receivable and payable data), financial entity performance research (answering time-bound questions from public filings and market data), and business brief generation (synthesising multi-source company intelligence reports). To reflect real deployment conditions, we evaluate our purpose-built agent, as well as the general-purpose agentic systems, under common tool access and latency-bounded settings. Results show that general-purpose agentic systems do not consistently meet finance-domain quality requirements under operational constraints, while the purpose-built Finance Agent for Microsoft 365 Copilot is more reliable across dimensions. We release the dataset, rubrics, harness, and analysis code as open-source to support reproducible comparison and adaptation to other enterprise finance environments.

Figures

Figures reproduced from arXiv: 2607.19409 by Ademola Okerinde, Jeremy Reynolds, Kidus Admassu, Said Bleik, Sarah Panda, Wolfgang M. Pauli.

Figure 1
Figure 1. Figure 1: Finance Agent Benchmark scores across eight evaluation dimensions for Finance Agent (purpose-built), [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Finance Agent accuracy by ERP scenario for financial obligation research ( [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-task-type benchmark scores for all four evaluated systems across the applicable evaluation dimensions. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of timeout enforcement on benchmark scores for GPT 5.5, Claude Haiku 4.5, and Claude Opus 4.7. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Mean score difference ∆ = s GPT-5.2 − s Opus 4.6 per (task-type, metric) cell. Stars denote statistical significance after Benjamini–Hochberg correction; ‡ marks cells exceeding the practical significance threshold (|∆| ≥ 0.05). Task-type labels: Business Brief = business brief generation, Finance Qa = financial entity performance research, Erp Qa = financial obligation research. higher call volume directl… view at source ↗
Figure 6
Figure 6. Figure 6: Spearman rank correlation ρ between GPT-5.2 and Claude Opus 4.6 scores per (task-type, metric) cell. High ρ indicates agreement on relative quality ordering independent of any level shift. The business brief citations cell is undefined (—) because GPT-5.2 assigned identical scores to all responses (zero variance). Finance Agent is excluded as its purpose-built pipeline has different latency characteristics… view at source ↗
Figure 7
Figure 7. Figure 7: Score distributions (left) and Bland–Altman plots (right) for the two lowest-agreement cells: [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Score distributions (left) and Bland–Altman plots (right) for the two lowest-agreement cells: [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of tool calls per question by provider and task type. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Inference latency distributions (violin plots with embedded box plots) for GPT 5.5, Claude Haiku 4.5, and [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 11 linked inside Pith

  1. [1]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Pith/arXiv arXiv 2009

  2. [4]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022a

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022a. Peng Yu, Wenyuan Liu, Yifeng Yang, Junyi Li, Zhe Zhang, and Xinyu Feng. Benchmarking llm tool-use in the wild. arXiv preprint arXiv:2604.06185,

    Pith/arXiv arXiv

  3. [8]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073,

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073,

    Pith/arXiv arXiv

  4. [9]

    Financebench: A new benchmark for financial question answering.arXiv preprint arXiv:2311.11944,

    14 FORCE-BenchTECHNICALREPORT Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering.arXiv preprint arXiv:2311.11944,

    Pith/arXiv arXiv

  5. [10]

    Bloomberggpt: A large language model for finance.arXiv preprint arXiv:2303.17564,

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, Gideon Mann, et al. Bloomberggpt: A large language model for finance.arXiv preprint arXiv:2303.17564,

    Pith/arXiv arXiv

  6. [11]

    Finben: A holistic financial benchmark for large language models.arXiv preprint arXiv:2402.12659,

    Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, et al. Finben: A holistic financial benchmark for large language models.arXiv preprint arXiv:2402.12659,

    Pith/arXiv arXiv

  7. [12]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. InAdvances in Neural Information Processing Systems (NeurIPS), 2022b. Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Arthur Szlam. Scienceworld: Is your agent smarter than a 5th grader? InProceedings of the 2...

    2022

  8. [13]

    Finance agent benchmark: Benchmarking llms on real-world financial research tasks.arXiv preprint arXiv:2508.00828,

    Antoine Bigeard, Langston Nashold, Rayan Krishnan, and Shirley Wu. Finance agent benchmark: Benchmarking llms on real-world financial research tasks.arXiv preprint arXiv:2508.00828,

    Pith/arXiv arXiv

  9. [15]

    Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714,

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714,

    Pith/arXiv arXiv

  10. [1986]

    15 FORCE-BenchTECHNICALREPORT Appendix A LLM Judge Bias Analysis To assess whether choice of LLM judge influences benchmark conclusions, we evaluated the same set of agent responses under two judges: GPT-5.2 (primary) and Claude Opus 4.6 (validation). The analysis was conducted on the Claude Haiku 4.5 inference run, which recorded the lowest accuracy scor...

    2023

  11. [2021]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  12. [2022]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, et al. Agentbench: Evaluating llms as agents. In Proceedings of the 2024 International Conference on Learning Representations (ICLR),

    2024

  13. [2023]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. InProceedings of the 2016 conference on empirical methods in natural language processing, pages 2383–2392,

    2016

  14. [2024]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045,

    Shunyu Yao, Noah Shinn, Poupak Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045,

    Pith/arXiv arXiv

  15. [2025]

    Finagentbench: A benchmark dataset for agentic retrieval in financial question answering.arXiv preprint arXiv:2508.14052,

    Chanyeol Choi, Jihoon Kwon, Alejandro Lopez-Lira, Chaewoon Kim, Minjae Kim, Juneha Hwang, Jaeseon Ha, Hojun Choi, et al. Finagentbench: A benchmark dataset for agentic retrieval in financial question answering.arXiv preprint arXiv:2508.14052,

    arXiv

  16. [2026]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Pith/arXiv arXiv