DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making

Raffi Khatchadourian

REVIEW 3 major objections 4 minor 15 references

A financial AI agent can return the same decision through very different tool paths, and outcome-only evaluation hides that instability.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:45 UTC pith:WLMEZUTZ

load-bearing objection Useful protocol, careful honesty, but the headline prospective gap may be partly sampling noise; the controlled rows are the real evidence. the 3 major comments →

arxiv 2607.20491 v2 pith:WLMEZUTZ submitted 2026-06-10 cs.AI cs.CLcs.LG

DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making

Raffi Khatchadourian This is my paper

classification cs.AI cs.CLcs.LG

keywords agent instabilityreplay evaluationtool-path agreementdecision agreementfinancial AIobservabilitydeterminismbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that outcome-only evaluation of financial AI agents is blind to process variation: an agent can return the same decision while changing which tools it calls, in what order, and with what arguments. Across a prospective extension of 190 eligible replay groups, decisions agree 94.2–95.1% of the time but exact tool-name paths agree only 66.9–69.4%, a 25.8–27.3-point gap; when arguments and result hashes are included, agreement drops to 45.0–51.5%. Even groups where every replay made the same decision showed path variation in 66.7–68.9% of cases under task weighting. The paper's point is that replay and change control need to measure the observable journey, not just the final label.

Core claim

The central discovery is a paired measurement gap: on the same eligible replay groups, decision agreement (DAR) and tool-path agreement (TAR) diverge substantially. In the prospective API extension, DAR is 94.2–95.1% while TARseq is 66.9–69.4%; the strong path that includes canonical arguments and deterministic result hashes agrees only 45.0–51.5%. The gap is not an artifact of one provider: the same pattern appears in retrospective API configurations with gaps of 5.0–21.0 points, while four pinned local configurations with an eight-replay harness stay within 0.1 points. The paper frames this as execution faithfulness under replay—fidelity of observable tool use—distinct from answer correctn

What carries the argument

The DFAH protocol: a replay-eligibility step that qualifies only groups with identical fixed configuration and present required channels, followed by modal agreement measures DAR (decision agreement) and TAR (tool-path agreement) computed over the same retained episodes. Path projections range from ordered tool-name sequences to multisets and sets, and in the prospective capture a strong fingerprint of ordered tool names, canonical arguments, and deterministic result hashes. The key work is aligning the denominators and distinguishing an observed empty path from a missing required channel, so that process variation is measured on exactly the same cases as decision agreement.

Load-bearing premise

The load-bearing premise is that the synthetic compliance and DataOps cases represent real financial agent workflows; if they don't, the agreement rates and gaps may not transfer, and the paper itself excludes accuracy claims because the labels lack independent expert adjudication.

What would settle it

A direct replication on independently audited real financial workflow logs: if exact tool-name path agreement on unanimous-decision groups exceeds 95%, the claim that process variation is hidden behind stable decisions would be falsified for real settings.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Outcome-only scores can report high accuracy while the underlying tool use is unstable; pairing DAR with TAR is necessary to see the process.
Unanimous decisions do not imply a stable process; change review should inspect path variation even when the final label never changes.
Replay-based benchmarking needs to pin the configuration (model, provider, sampling controls, tool schemas, data) and record arguments and result hashes, not just tool names.
Operational workflows should pre-register an investigation trigger—such as a unanimous decision with multiple paths—and use shadow replays on a risk-stratified sample rather than blocking every live case.
The observed gaps are diagnostic, not verdicts: variation is a signal for inspection, and zero gap does not mean the path is correct.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the synthetic compliance and DataOps tasks are representative, real financial workflows with richer tool sets and stateful side effects may show even larger decision–path gaps; this is an extrapolation, not a paper claim.
A semantic trajectory-equivalence score—treating near-miss paths as operationally equivalent—could shrink the gap, but the paper explicitly avoids this; testing whether such a score changes the findings would clarify which path differences actually matter.
The near-perfect agreement of the pinned local harness suggests that much of the API variation may come from provider-side sampling or unrecorded settings rather than inherent model nondeterminism; an interventional study that toggles provider sampling parameters could separate these causes.
Extending DFAH to non-financial agent tasks (coding, email, customer service) could reveal whether the decision–path blind spot is a general property of tool-using agents or specific to these financial workflows.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful protocol, careful honesty, but the headline prospective gap may be partly sampling noise; the controlled rows are the real evidence.

read the letter

Jake — quick take. This paper is worth reading for anyone who evaluates financial agents. It makes a real point: outcome-only scores miss process variation, and it gives a clean protocol for measuring decision vs. tool-path agreement on the same eligible replays. The eligibility logic (missing channels don't count as agreement or empty paths) is thoughtful. The paper is also unusually honest about post-run corrections, a parser fallback, and the fact that the synthetic labels aren't adjudicated. That earns real credit.

The central empirical result, though, splits into two parts. The controlled rows are solid: four local configurations at temperature zero, seeded, show near-zero gaps; Gemini 2.0 Flash at temp zero without a seed shows a 5.0-point gap. That's a defensible, modest effect. The prospective API extension — the one that produces the 25.8–27.3-point headline gaps — has a problem. The manifests explicitly record 'omitted sampling parameters.' That means the three replays per case aren't necessarily identical executions; they're draws from the provider default. Under a stochastic sampling regime, low path agreement and high decision agreement is exactly what you'd expect. The paper's own comparability contract in Appendix B requires equality of decoding controls, and this extension doesn't meet it. So the abstract's headline numbers shouldn't be presented as 'with input and settings fixed.' They're sensitivity evidence at best. The paper does flag the omission, but only in the setup section, not in the abstract or the conclusion. That's the biggest soft spot.

The other soft spots are minor relative to that: the tasks are synthetic, sparse, and lack dual-expert adjudication, so accuracy isn't analyzed; the Flash effect is carried largely by the DataOps cases, not compliance; the permutation tests are finite-corpus, not population inference. All of these are disclosed. The omitted sampling parameters is the one thing that actually threatens the headline claim.

My take: the protocol and the controlled results deserve a serious referee. The authors should either pin and record sampling parameters in the prospective extension, or explicitly label those numbers as provider-default sensitivity evidence rather than a test of identical replays. If they do that, this is a useful contribution to agent evaluation.

Referee Report

3 major / 4 minor

Summary. The paper introduces DFAH-Bench, a replay-based benchmark that separates outcome stability from process stability for financial AI agents. It defines decision agreement (DAR) and tool-path agreement (TAR) over paired, eligibility-qualified replays, and measures these across a retrospective corpus (4,157 episodes, 719 groups) and a prospective API extension (570 eligible episodes, 190 groups). The headline finding is that decisions remain stable (94.2–95.1% agreement) while exact tool-name paths vary substantially (66.9–69.4%), producing 25.8–27.3-point gaps, and that path variation persists even among unanimous-decision groups (66.7–68.9%). The paper is unusually transparent: it discloses post-run corpus corrections, a legacy parser fallback, a temperature-recording defect, a one-group miss on a predeclared minimum, and finite-corpus sensitivity intervals rather than population confidence bounds. It also provides code, synthetic fixtures, hashes, and regeneration scripts.

Significance. The distinction between a repeated decision and a repeated execution process is practically important for financial AI governance, replay, audit, and change control. The methodological core—pairing DAR and TAR on the same denominator, using modal agreement, defining eligibility and required channels, and separating sequence/multiset/set abstractions—is sound and well described. The paper ships reproducible artifacts and explicitly avoids overclaiming accuracy or safety. However, the headline prospective result, which supplies the abstract's central numbers, is compromised by the omission of sampling parameters, making the reported gap potentially an artifact of stochastic decoding rather than agent process variation. The controlled retrospective rows show a much smaller gap (5.0 points for Gemini Flash), so the core claim survives in weaker form but needs re-scoping or additional experiments.

major comments (3)

[§4, Appendix B, Table 5] The prospective API extension's manifests record 'omitted sampling parameters' while Appendix B's comparability contract requires equality of decoding controls. The extension's three replays are therefore not identical executions with fixed settings, contradicting RQ1's 'with input and settings fixed.' Under stochastic decoding, three draws can share a modal label while differing in tool choices, so the 25.8–27.3-point gap conflates process variation with decoding noise. The paper's own controlled Gemini Flash row shows only a 5.0-point gap. This is load-bearing because the abstract and conclusion headline the prospective extension. The fix is to pin and record sampling parameters (temperature, top-p, seed) for all replays, or to downgrade the extension to sensitivity evidence and revise the abstract accordingly.
[§3.1 vs §4] Section 3.1 defines a configuration as including 'sampling controls,' but the prospective extension records 'omitted sampling parameters.' Thus the extension does not hold the configuration fixed, and the 'identical replays' language is inaccurate. If the intended claim is about provider-default deployment behavior, that should be stated explicitly; the current text wavers between 'identical settings' and 'deployment configuration,' allowing the reader to infer that the large gap reflects agent instability when it may reflect unrecorded stochasticity.
[§5, RQ2 unanimous-decision analysis] The claim that 'even among unanimous-decision groups, paths vary in 66.7–68.9% under task weighting' comes from the same prospective extension with omitted sampling parameters. Unanimity in the modal decision across three stochastic draws is not evidence that the same decision would be reached under fixed settings. This conditional analysis inherits the same confounding. The retrospective controlled rows (e.g., 24.7% for Gemini Flash) provide cleaner evidence for RQ2, so the prospective unanimous-decision statistic should not be presented as a headline without sampling-control guarantees.

minor comments (4)

[Abstract/§1] The abstract says 'changing the tools, order, or evidence used'; in the retrospective corpus, only tool names and order are captured, not evidence content. Using 'evidence' loosely could overstate what the retrospective measures actually show. Consider distinguishing 'tool names and order' from 'arguments and results' consistently.
[§2] The sentence 'may both returnescalate' has a missing space; minor typographical issue.
[§4, 'Corpus corrections'] The paper states the legacy parser could substitute the final ontology label after extraction failure without recording provenance. The stress bounds in §8 are appreciated, but the sentence 'It may inflate DAR and the paired gap' should clarify that the direction is unknown for any specific comparison; the paper later says this, but the earlier phrasing could be read as an assumption.
[Appendix C.1] The definition of DCB uses H(p)/log 3; it would help to state the base of the logarithm explicitly (natural log or log2), though the normalization is invariant up to a constant.

Circularity Check

0 steps flagged

No circular derivation: DAR and TAR are distinct measured observables; no fitted input is renamed as a prediction, and no load-bearing self-citation appears.

full rationale

The paper's derivation is self-contained at the measurement level. DAR (Eq. 1) and TARseq (Eq. 2) are modal agreement fractions over two distinct projections of the same replay corpus (decisions vs. ordered tool-name paths), and the gap (Eq. 3) is a paired contrast between those marginal concentrations. It is not a fitted parameter later relabeled as a prediction: no parameter is estimated from one subset and then used to 'predict' a closely related quantity. Modal agreement is computed from all eligible episodes, and the reported sensitivity intervals are transparently labeled as finite-corpus resampling summaries, not population inference. TARstrong is intentionally stricter than TARseq by including canonical arguments and result hashes, so its being lower is a construct-monotonicity fact; however, the reported magnitudes (51.5% and 45.0%) are empirical counts from the prospective extension, not consequences of the definitions alone. The paper also acknowledges the main non-circular validity threat: the prospective API extension manifests record 'omitted sampling parameters,' so provider-default stochastic decoding may confound the 'settings fixed' claim. That is a correctness/external-evidence concern, not a circular step, because the DAR-TAR gap is not true by construction and the paper itself separates temperature-zero Gemini rows and labels Claude rows as sensitivity evidence. No load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation chains appear; cited prior work on repeat variation is contextual. The public repository regenerates tables from retained compact logs, so the reported numbers are backed by external artifacts rather than by the metric definitions.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

No numbers are fitted to data in the central derivation; the statistics are computed modal fractions. The main assumptions are domain-level: equality of paths is the right operational fidelity measure, and the synthetic cases and retrospective logs are sufficient. No new model, particle, force, or entity is postulated.

axioms (3)

domain assumption Exact equality of tool-name paths is the right granularity for measuring replay fidelity.
Introduced in Section 3.2; the paper explicitly calls this a deliberately conservative choice and anticipates future semantic trajectory scores.
domain assumption The published per-model agreement rates are meaningful given the small, synthetic, non-adjudicated case sets.
Section 8 and Appendix A state the cases are synthetic and sparse, no dual-expert annotation has been completed, and the portfolio fixture was excluded; this limits whether per-model rates are meaningful.
domain assumption The retrospective logs are sufficient to reconstruct the reported aggregate measures despite the legacy parser fallback.
Section 8 states the logs do not identify fallback episodes, so any effect on DAR is not recoverable from the data.

pith-pipeline@v1.3.0-alltime-deepseek · 13118 in / 5315 out tokens · 45453 ms · 2026-08-02T11:45:10.492664+00:00 · methodology

0 comments

read the original abstract

A financial AI agent can repeat a decision while changing the tools, order, or recorded arguments and results used to reach it. Outcome-only evaluation misses this variation, even when it matters for replay and change control. DFAH-Bench operationalizes the Determinism-Faithfulness Assurance Harness (DFAH), where faithfulness means fidelity of observable execution under replay, not answer correctness. The protocol qualifies comparable, sufficiently observed replays and measures decision agreement (DAR) and tool-path agreement (TAR) over the same eligible groups. We analyze 4,157 retrospective episodes from configurations with observed tool use across 719 synthetic compliance and financial DataOps groups, together with an argument-aware prospective extension comprising 570 eligible episodes across 190 groups. In that extension, decisions agree 94.2-95.1% while exact tool-name paths agree 66.9-69.4%, producing 25.8-27.3 percentage-point gaps; argument-and-result trajectory agreement falls to 45.0-51.5%. Even among unanimous-decision groups, paths vary in 66.7-68.9% under task weighting. DFAH-Bench makes the execution behind a stable decision visible for replay, investigation, and change review.

Figures

Figures reproduced from arXiv: 2607.20491 by Raffi Khatchadourian.

**Figure 2.** Figure 2: Argument-aware prospective API extension. Labels give eligible groups [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Figure 3: Corrected task-level decision–path gaps. The excluded portfolio fixture does not appear. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

**Figure 4.** Figure 4: Modal-decision concentration across cases in the corrected compliance and DataOps slice. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages · 2 internal anchors

[1]

Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

Jan Batzner, Sree Harsha Nelaturu, Damian Stachura, Anastassia Kornilova, Jon Crall, Tommaso Cerruti, Yanan Long, Yifan Mai, Sanchit Ahuja, Asaf Yehudai, et al. Every eval ever: A unifying schema and community repository for AI evaluation results. arXiv preprint arXiv:2606.14516, 2026. doi:10.48550/arXiv.2606.14516

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2606.14516 2026
[2]

Revised guidance on model risk management: Supervisory letter SR 26-2

Board of Governors of the Federal Reserve System . Revised guidance on model risk management: Supervisory letter SR 26-2 . Supervisory letter, 2026. URL https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm

2026
[3]

Yupeng Cao, Haohang Li, Weijin Liu, Wenbo Cao, Anke Xu, Lingfei Qian, Xueqing Peng, Minxue Tang, Zhiyuan Yao, Jimin Huang, K. P. Subbalakshmi, Zining Zhu, Jordan W. Suchow, and Yangyang Yu. FinTrace : Holistic trajectory-level evaluation of LLM tool calling for long-horizon financial tasks. arXiv preprint arXiv:2604.10015, 2026. doi:10.48550/arXiv.2604.10015

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.10015 2026
[4]

Visibility into AI agents

Alan Chan, Carson Ezell, Max Kaufmann, Kevin Wei, Lewis Hammond, Herbie Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, Lennart Heim, and Markus Anderljung. Visibility into AI agents. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 958--973. ACM, 2024. doi:10.1145/3630106.3658948

arXiv 2024
[6]

TRAJECT-Bench : A trajectory-aware benchmark for evaluating agentic tool use

Pengfei He, Zhenwei Dai, Bing He, Hui Liu, Xianfeng Tang, Hanqing Lu, Juanhui Li, Jiayuan Ding, Subhabrata Mukherjee, Suhang Wang, Yue Xing, Jiliang Tang, and Benoit Dumoulin. TRAJECT-Bench : A trajectory-aware benchmark for evaluating agentic tool use. arXiv preprint arXiv:2510.04550, 2025

arXiv 2025
[7]

AI safety best practices for regulated environments

LF AI & Data Security and Compliance Working Group . AI safety best practices for regulated environments. White paper Version 1.0, LF AI & Data Foundation, June 2026. URL https://lfaidata.foundation/wp-content/uploads/sites/3/2026/06/AI_Safety_Best_Practices.pdf

2026
[8]

A review of evaluation metrics for text similarity

Mingshu Li and Dhagash Mehta. A review of evaluation metrics for text similarity. SSRN working paper, June 2026. URL https://ssrn.com/abstract=6900601

2026
[9]

ToolSandbox : A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities

Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. ToolSandbox : A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160--1183. Associ...

doi:10.18653/v1/2025.findings-naacl.65 2025
[10]

ReasonBENCH : Benchmarking the (in)stability of LLM reasoning

Nearchos Potamitis, Vansh Ramani, Har Ashish Arora, Dhairya Kuchhal, Lars Klein, and Akhil Arora. ReasonBENCH : Benchmarking the (in)stability of LLM reasoning. arXiv preprint arXiv:2512.07795, 2025. doi:10.48550/arXiv.2512.07795

Pith/arXiv arXiv doi:10.48550/arxiv.2512.07795 2025
[11]

FinResearchBench : A logic tree based agent-as-a-judge evaluation framework for financial research agents

Rui Sun, Zuo Bai, Wentao Zhang, Yuxiang Zhang, Li Zhao, Shan Sun, and Zhengwen Qiu. FinResearchBench : A logic tree based agent-as-a-judge evaluation framework for financial research agents. In Proceedings of the 6th ACM International Conference on AI in Finance, pages 656--664. ACM, 2025. doi:10.1145/3768292.3770364

arXiv 2025
[12]

Scalable runtime governance for agentic AI in financial services

Lukasz Szpruch, Agus Sudjianto, Tanveer Bhatti, and Gary Ang. Scalable runtime governance for agentic AI in financial services. SSRN Electronic Journal, 2026. doi:10.2139/ssrn.6567199

doi:10.2139/ssrn.6567199 2026
[13]

Assessing consistency and reproducibility in the outputs of large language models: Evidence across diverse finance and accounting tasks

Julian Junyan Wang and Victor Xiaoqi Wang. Assessing consistency and reproducibility in the outputs of large language models: Evidence across diverse finance and accounting tasks. arXiv preprint arXiv:2503.16974, 2025. doi:10.48550/arXiv.2503.16974

Pith/arXiv arXiv doi:10.48550/arxiv.2503.16974 2025
[14]

Self-consistency improves chain of thought reasoning in language models

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023

2023
[15]

FinBen : A holistic financial benchmark for large language models

Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. FinBen : A holistic financial benchmark for large language models. In Advances in Neural Information Processing Systems: Datasets and Benchmarks Track, 2024

2024
[16]

-bench : A benchmark for tool-agent-user interaction in real-world domains

Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. -bench : A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations, 2025

2025

[1] [1]

Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

Jan Batzner, Sree Harsha Nelaturu, Damian Stachura, Anastassia Kornilova, Jon Crall, Tommaso Cerruti, Yanan Long, Yifan Mai, Sanchit Ahuja, Asaf Yehudai, et al. Every eval ever: A unifying schema and community repository for AI evaluation results. arXiv preprint arXiv:2606.14516, 2026. doi:10.48550/arXiv.2606.14516

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2606.14516 2026

[2] [2]

Revised guidance on model risk management: Supervisory letter SR 26-2

Board of Governors of the Federal Reserve System . Revised guidance on model risk management: Supervisory letter SR 26-2 . Supervisory letter, 2026. URL https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm

2026

[3] [3]

Yupeng Cao, Haohang Li, Weijin Liu, Wenbo Cao, Anke Xu, Lingfei Qian, Xueqing Peng, Minxue Tang, Zhiyuan Yao, Jimin Huang, K. P. Subbalakshmi, Zining Zhu, Jordan W. Suchow, and Yangyang Yu. FinTrace : Holistic trajectory-level evaluation of LLM tool calling for long-horizon financial tasks. arXiv preprint arXiv:2604.10015, 2026. doi:10.48550/arXiv.2604.10015

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.10015 2026

[4] [4]

Visibility into AI agents

Alan Chan, Carson Ezell, Max Kaufmann, Kevin Wei, Lewis Hammond, Herbie Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, Lennart Heim, and Markus Anderljung. Visibility into AI agents. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 958--973. ACM, 2024. doi:10.1145/3630106.3658948

arXiv 2024

[5] [6]

TRAJECT-Bench : A trajectory-aware benchmark for evaluating agentic tool use

Pengfei He, Zhenwei Dai, Bing He, Hui Liu, Xianfeng Tang, Hanqing Lu, Juanhui Li, Jiayuan Ding, Subhabrata Mukherjee, Suhang Wang, Yue Xing, Jiliang Tang, and Benoit Dumoulin. TRAJECT-Bench : A trajectory-aware benchmark for evaluating agentic tool use. arXiv preprint arXiv:2510.04550, 2025

arXiv 2025

[6] [7]

AI safety best practices for regulated environments

LF AI & Data Security and Compliance Working Group . AI safety best practices for regulated environments. White paper Version 1.0, LF AI & Data Foundation, June 2026. URL https://lfaidata.foundation/wp-content/uploads/sites/3/2026/06/AI_Safety_Best_Practices.pdf

2026

[7] [8]

A review of evaluation metrics for text similarity

Mingshu Li and Dhagash Mehta. A review of evaluation metrics for text similarity. SSRN working paper, June 2026. URL https://ssrn.com/abstract=6900601

2026

[8] [9]

ToolSandbox : A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities

Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. ToolSandbox : A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160--1183. Associ...

doi:10.18653/v1/2025.findings-naacl.65 2025

[9] [10]

ReasonBENCH : Benchmarking the (in)stability of LLM reasoning

Nearchos Potamitis, Vansh Ramani, Har Ashish Arora, Dhairya Kuchhal, Lars Klein, and Akhil Arora. ReasonBENCH : Benchmarking the (in)stability of LLM reasoning. arXiv preprint arXiv:2512.07795, 2025. doi:10.48550/arXiv.2512.07795

Pith/arXiv arXiv doi:10.48550/arxiv.2512.07795 2025

[10] [11]

FinResearchBench : A logic tree based agent-as-a-judge evaluation framework for financial research agents

Rui Sun, Zuo Bai, Wentao Zhang, Yuxiang Zhang, Li Zhao, Shan Sun, and Zhengwen Qiu. FinResearchBench : A logic tree based agent-as-a-judge evaluation framework for financial research agents. In Proceedings of the 6th ACM International Conference on AI in Finance, pages 656--664. ACM, 2025. doi:10.1145/3768292.3770364

arXiv 2025

[11] [12]

Scalable runtime governance for agentic AI in financial services

Lukasz Szpruch, Agus Sudjianto, Tanveer Bhatti, and Gary Ang. Scalable runtime governance for agentic AI in financial services. SSRN Electronic Journal, 2026. doi:10.2139/ssrn.6567199

doi:10.2139/ssrn.6567199 2026

[12] [13]

Assessing consistency and reproducibility in the outputs of large language models: Evidence across diverse finance and accounting tasks

Julian Junyan Wang and Victor Xiaoqi Wang. Assessing consistency and reproducibility in the outputs of large language models: Evidence across diverse finance and accounting tasks. arXiv preprint arXiv:2503.16974, 2025. doi:10.48550/arXiv.2503.16974

Pith/arXiv arXiv doi:10.48550/arxiv.2503.16974 2025

[13] [14]

Self-consistency improves chain of thought reasoning in language models

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023

2023

[14] [15]

FinBen : A holistic financial benchmark for large language models

Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. FinBen : A holistic financial benchmark for large language models. In Advances in Neural Information Processing Systems: Datasets and Benchmarks Track, 2024

2024

[15] [16]

-bench : A benchmark for tool-agent-user interaction in real-world domains

Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. -bench : A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations, 2025

2025