Pith. sign in

REVIEW 4 major objections 4 minor 49 references

FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new finance benchmark spans the full investor workflow and shows that the tool harness, not the model alone, drives agent quality and cost.

desk verdict A well-curated, genuinely useful finance-agent benchmark whose headline rankings I would not trust yet, because the LLM judges never verify rubrics against their own sources and no human validation is reported. read the letter →

arxiv 2608.11683 v1 pith:I3M7YRWF submitted 2026-08-12 cs.AI cs.CL

Yuhao Zhang , O. Ozan Koyluoglu , Thejas Venkatesh , Richard Diehl Martinez , Vishank Bhatia , Arash Alidoust , Ashwin Paranjape This is my paper · ORCID
classification cs.AIcs.CL
keywords financeagentsbenchmarkrubric-basedevaluationLLM-as-a-judgeinvestorworkflowopen-endedresearchagentharness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FrontierFinance is a publicly released benchmark of 220 expert-written investment-research queries, graded by 11,543 binary rubrics, each tied to a public data source. The paper's aim is to measure what existing finance benchmarks do not: whether an AI agent can plan, search, synthesize, and write a long-form research answer across the entire investor workflow rather than extract a single number. The central claim is that the benchmark is both broader and harder than current public finance benchmarks, and that when frontier models are tested under a shared harness, the surrounding tool harness changes quality and cost more than the underlying model does. A sympathetic reader would take away that open-ended financial research remains largely unsolved: the best system reaches 56% overall, and the hardest use cases, screening and discovery and sector/industry/macro, top out near 33% and 39%.

What carries the argument

The central scoring object is the rubric qualification rate: each query is decomposed into expert-authored binary rubrics, and a system's per-query score is the fraction of those rubrics that a majority of three independent LLM judges finds satisfied in the agent's long-form report. Rubrics carry source attribution to a public data source tier, which turns open-ended answers into checkable criteria. The second load-bearing mechanism is the consensus Bradley–Terry difficulty scale, which places queries and whole benchmark suites on a shared 'how hard is this query' axis and supports the claim that FrontierFinance is harder than its predecessors.

What would settle it

Take a random sample of rubric verdicts from the released grading code and have finance experts judge whether each report actually satisfies the rubric and whether the supporting fact is true in the attributed public source; if expert agreement with the LLM majority is substantially below the reported inter-judge consistency, the difficulty comparison and system rankings lose their evidential value.

Watch

Extended reading notes

Core claim

The paper's central discovery is twofold. First, the agent harness—the suite of tools, prompts, and orchestration surrounding the LLM—strongly shapes both answer quality and cost: the authors' in-house system leads at 56.0%, ahead of the strongest frontier model deployed under an open-source finance harness (49.2%) at roughly 2.2x lower cost, and the best open-weight model reaches 46.4%, nearly matching the best proprietary model while costing about 4.5x less. Second, the two most open-ended use cases—screening and discovery and sector/industry/macro—remain hardest across every system tested, with the best systems scoring only 33% and 39%. The paper further claims, using a consensus Bradley–Terry difficulty model fit over roughly 77,000 pairwise judgments, that FrontierFinance is substantially harder and wider in difficulty than three comparable public benchmarks, which cluster in the easy-to-medium range.

Load-bearing premise

The scoring pipeline rests on the assumption that three LLM judges, voting by majority and checking only whether the report contains each rubric's statement, correctly decide which rubrics are satisfied, without any human-verified check of those facts against the attributed public source.

Editorial extensions

If this is right

  • Benchmark comparisons that mix model and harness cannot be read as model rankings; the same model will place very differently depending on the tools and orchestration it is given.
  • Open-weight models at roughly 46% qualification for under $1 per query put strong price-performance pressure on proprietary APIs, and the paper shows the gap closing within about two months of model release.
  • The two open-ended use cases, screening and discovery and sector/industry/macro, define the clearest headroom: even the best systems leave most of their rubrics unsatisfied.
  • The three-phase tool-use pattern—data gathering, mid-rollout synthesis, and answer preparation—appears across all systems, suggesting task structure drives agent behavior more than any single model policy.
  • Agents that navigate directly to remembered financial URLs rather than discovering sources through search incur higher access-error rates and token waste, pointing to a concrete efficiency failure mode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the harness effect is real, raw model-API benchmarks for finance systematically underreport what a well-instrumented agent can do; the field should standardize the harness before comparing models.
  • The rubric-based design is vulnerable to format gaming: a system that learns to state rubric-like claims without verifying them against sources could inflate its score, so a human audit of judge verdicts against the attributed sources would be a natural validation step.
  • Because every query is date-anchored, scores will decay as future models memorize post-date data; periodic re-annotation with fresh query dates is the natural maintenance path, and the paper's reserved internal query pool is a ready reservoir for that.
  • The Bradley–Terry difficulty score could be reused to predict which queries benefit most from extra reasoning effort or tool budget, since the paper finds diminishing returns beyond each model's default effort.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents FrontierFinance, a benchmark of 220 expert-authored financial research queries and 11,543 binary rubrics spanning six use cases across an investor workflow. Queries are timestamped, rubrics are tagged by essentiality, content category, and expected data source, and answers are scored by a macro-averaged rubric qualification rate computed by majority vote of three LLM judges. The authors evaluate frontier proprietary and open-weight models under three harnesses—a minimal web-search harness, an open-source Finance Agent v2 harness, and Samaya's in-house harness—and report that the harness strongly shapes quality and cost, with Samaya's system at 56.0%, Claude Fable 5 at 49.2%, and Kimi K3 at 46.4%. A trajectory analysis identifies common three-phase tool-use patterns and a link between parametric-knowledge URL recall and higher parse errors.

Significance. If the measurement pipeline is valid, FrontierFinance is a valuable resource: it is among the largest open finance-agent benchmarks to date, covers workflow breadth absent from prior suites, provides expert-authored rubrics with source labels, timestamps queries to reduce temporal leakage, and releases grading code for reproducibility. The trajectory analysis around tool-use efficiency and URL errors is a useful methodological contribution. The central difficulty and ranking conclusions are, however, contingent on the validity of LLM-judge scoring, which the paper does not establish with human validation or source verification.

major comments (4)
  1. [5.1, Appendix F] The headline metric R in Eq. (3) is computed from majority verdicts of three LLM judges, but the paper reports no human agreement study for these judges; the only calibration statement is that the majority 'closely matched a larger committee of nine judges' in preliminary experiments, without statistics. The judge prompt in Appendix F asks whether the report 'adequately satisfies' each rubric and never consults the rubric's source attribution, so a report containing the rubric's wording can be scored as satisfied even if the underlying factual claim is not verified against the attributed public source. Because R_all and R_must-have mediate the difficulty comparison, the harness-effect claim, and the Samaya-versus-frontier ranking in Table 4, this is a measurement-validity issue that affects the paper's central conclusions. A human-annotated validation subset with per-rubric agreement, or a judge protocol that verifies claims against the attributed sources, is needed before the reported numbers can support the claims.
  2. [4.2, Appendix C] The difficulty scale used to claim FrontierFinance is harder than existing benchmarks (Figure 5) is itself derived from LLM pairwise judgments, with robustness checks (Tables 10-11, Figure 12) that only show consistency across LLM judges and not agreement with expert or human difficulty ratings. In addition, the public 220-query set was constructed by stratifying on this difficulty score, so the monotonically decreasing agent performance across easy/medium/hard buckets in Table 2 is at least partly a consequence of the selection procedure rather than an independent validation of the scale. Please provide expert judgment on a sample of pairs or another external anchor, and report an analysis of the public subset that accounts for the stratification.
  3. [6, Table 4] Several headline comparisons are reported without uncertainty estimates. For example, Kimi K3 (46.4%) is only 0.4 pp behind GPT 5.6 Sol (46.8%), and the Screening & Discovery row of Table 16 is based on 17 queries; macro-averaging over so few queries can make differences of this magnitude indistinguishable from noise. The paper should report bootstrap confidence intervals, significance tests, or per-query score distributions before drawing conclusions about the ordering of systems and the claim that open-weight models 'nearly match' proprietary ones.
  4. [4.1, Table 9] The dataset is described as 'source-attributed,' but the source taxonomy includes 'professional knowledge' as a top-level category, accounting for 19% of rubrics overall and 32% for Sector, Industry & Macro. These rubrics are not tied to a specific public source, which weakens both the objectivity of rubric scoring and the claim of full source attribution; the paper should either reclassify such rubrics or explicitly discuss how they are verified.
minor comments (4)
  1. [Appendix F] The system prompt for the grading judge ends mid-sentence at 'Likewise against "7.14%' with no continuation; the full prompt must be included for the claimed reproducibility.
  2. [5.1] No details are given for the 'preliminary experiments' with the nine-judge committee; report the agreement statistic and sample size.
  3. [6, Figure 7] The statement that the harness ordering 'holds across all six use cases and rubric categories' should be qualified: Table 16 shows small within-harness differences for Screening & Discovery, and the claim would benefit from an explicit statistical test rather than the radar chart alone.
  4. [3.1] The four-stage pipeline is described qualitatively; reporting inter-annotator agreement for rubric authoring and source labeling would strengthen the reliability claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivational circularity: the benchmark claims rest on independent measurements, and the only self-citation is non-load-bearing.

full rationale

FrontierFinance is an empirical benchmark paper, so the main circularity patterns—equations reducing to their inputs or fitted parameters renamed as predictions—do not apply. The difficulty score is a fitted Bradley–Terry latent variable estimated from LLM pairwise judgments, but the paper does not present this fit as a prediction; it externally anchors the scale against expert solve times on Finance Agent Benchmark v2 (Section 4.2: “the median time rises 20→40→45→60 minutes across BT score quartiles (Spearman ρ=0.67)”) and against independently measured agent qualification rates (Table 2). The rubric qualification rate is an operational definition of answer quality, and Appendix C reports stability analyses for judge noise, position bias, and judgment volume; those checks are not circular. The self-citation to the authors’ Criteria-Eval [26] is a provenance statement for a checklist methodology that is also supported by external citations [22–25] and by the released grading code, so it is not load-bearing. Section 8 explicitly concedes that “a system whose answer overlaps with the rubric may outscore one with an equally correct but differently framed response”; this is a measurement-validity limitation about LLM judging, not a circular derivation, and the skeptical concern that the judge prompt does not verify source attribution is likewise a validity threat rather than an equivalence-by-construction. No equation in the paper is equal to its input by construction, and the headline claims—the harness effect, difficulty ordering, and open-weight competitiveness—rest on independent measurements.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The ledger collects the tuning constants and background assumptions that the benchmark's difficulty and quality claims rest on. The key free parameters are in the Bradley-Terry difficulty model and the judge ensemble; the key assumptions are that expert rubrics are binary, that LLM judges are reliable, and that the released 220 queries represent the workflow.

free parameters (5)
  • BT regularization lambda = 1e-3
    Used in the confidence-weighted Bradley-Terry fit for query difficulty (Appendix C.1); its value was chosen by the authors and affects the difficulty terciles that support the 'harder than existing benchmarks' claim.
  • Margin weights c(1,2,3) = (0.3, 1.0, 1.3)
    Confidence weights for pairwise difficulty judgments in the BT model; chosen by the authors and used to discount close calls.
  • Tie threshold = 0.5
    Pairwise judgments are merged by signed-average with ties within 0.5; this threshold affects consensus verdicts and the resulting difficulty scores.
  • Judge ensemble = GPT 5.4, Gemini 3.1 Pro, Claude Sonnet 4.6
    The three judge models for rubric grading were chosen to match a larger nine-judge committee in preliminary experiments, but the agreement data are not reported.
  • Tool call limits = 200 calls, 300 seconds
    Imposed on both the adapted Finance Agent v2 harness and the Samaya harness; these limits affect answers and reported latency and cost.
assumptions (5)
  • domain assumption Expert-authored rubrics are binary and objectively checkable against public evidence
    The entire scoring pipeline assumes each rubric can be judged 0/1 without ambiguity; Section 8 concedes this fails for subjective use cases like screening and discovery.
  • domain assumption Majority verdict of three LLM judges is a valid measure of rubric satisfaction
    No human agreement data are provided; the judge prompt in Appendix F is the only specification, and it checks content presence, not factual truth.
  • domain assumption Bradley-Terry scores over five axes track real analyst difficulty
    Difficulty is defined via LLM pairwise judgments; validation against human solve times uses only 27 queries (Spearman 0.67), a modest basis for the claim that the benchmark is harder than existing ones.
  • domain assumption The 220 released queries represent the full investor workflow
    Queries are a stratified sample from an internal pool of over 4,500, selected by the authors; the selection criteria are described but not independently audited.
  • domain assumption Public data sources remain available and unchanged relative to each query date
    Rubrics are tied to public sources as of the query date; changes in availability or content could make rubrics unsatisfiable, though the paper excludes predictive queries to reduce this risk.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents." pith.science (2026). https://pith.science/paper/I3M7YRWF

@misc{pith2026260811683,
  author       = {Pith},
  title        = {Pith review of: FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I3M7YRWF}},
  note         = {Machine review of arXiv:2608.11683}
}
read the original abstract

AI agents are increasingly deployed for professional investment research, yet no benchmark captures the complexity of the full investor workflow. Existing benchmarks mainly target financial data extraction, a narrow slice that current models have largely saturated, while reference-based metrics and generic LLM-as-a-judge scoring fall short on the open-ended, long-form answers that real analyst queries demand. We introduce FrontierFinance, a fully open benchmark of 220 expert-crafted queries and 11,543 source-attributed rubrics spanning six crucial use cases across the full investor workflow. FrontierFinance is both broader and harder than existing public finance benchmarks. Evaluating frontier models and agent systems under a common harness restricted to publicly available data, we find that the tool harness, not the model alone, strongly shapes quality and efficiency; that Samaya's in-house system leads at 56.0%, ahead of the strongest frontier model (Claude Fable 5, 49.2%) at roughly 2.2x lower cost; and that the best open-weight model (Kimi K3, 46.4%) nearly matches the best proprietary model at 4.5x lower cost. Screening & Discovery and Sector, Industry & Macro remain the hardest use cases across all systems, where even the best systems reach only 33% and 39%. We make the dataset and grading code publicly available.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 24 canonical work pages

  1. [1]

    FinanceBench: A new benchmark for financial question answering, 2023

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. FinanceBench: A new benchmark for financial question answering, 2023. URL https://arxiv.org/abs/2311.11944

    arXiv 2023

  2. [2]

    FinQA: A 16 dataset of numerical reasoning over financial data

    ZhiyuChen,WenhuChen,ChareseSmiley,SameenaShah,IanaBorova,DylanLangdon,Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A 16 dataset of numerical reasoning over financial data. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, Online and Punta Cana, Dominican R...

    work page 2021

  3. [3]

    TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language...

    work page

  4. [4]

    ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering

    Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6279–6292, Abu Dhabi, United Arab Emirates, December 2022. Association ...

    doi:10.18653/v1/2022.emnlp-main.421 2022

  5. [5]

    DocFinQA: A long-context financial reasoning dataset

    Varshini Reddy, Rik Koncel-Kedziorski, Viet Dac Lai, Michael Krumdick, Charles Lovering, and Chris Tanner. DocFinQA: A long-context financial reasoning dataset. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 445–458, Bangkok, Thailand, August 2024. Association for Computational Lin...

    doi:10.18653/v1/2024.acl-short.42 2024

  6. [6]

    Fi- nanceReasoning: Benchmarking financial numerical reasoning more credible, comprehensive and challenging

    Zichen Tang, Haihong E, Ziyan Ma, Haoyang He, Jiacheng Liu, Zhongjun Yang, Zihua Rong, Rongjin Li, Kun Ji, Qing Huang, Xinyang Hu, Yang Liu, and Qianhe Zheng. Fi- nanceReasoning: Benchmarking financial numerical reasoning more credible, comprehensive and challenging. InProceedings of the 63rd Annual Meeting of the Association for Com- putational Linguisti...

    work page

  7. [7]

    FinBen: A holistic financial benchmark for large language models

    Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, Zhiyang Deng, Yuechen Jiang, Zhiyuan Yao, Haohang Li, Yangyang Yu, Gang Hu, Jiajia Huang, Xiao-Yang Liu, Alejandr...

    work page 2024

  8. [8]

    Financeagentbenchmark: Benchmarking LLMs on real-world financial research tasks, 2025

    AntoineBigeard,LangstonNashold,RayanKrishnan,andShirleyWu. Financeagentbenchmark: Benchmarking LLMs on real-world financial research tasks, 2025. URLhttps://arxiv. org/abs/2508.00828. 17

    arXiv 2025

Show all 49 references
  1. [9]

    Kim, Pratyush K

    Alex Wang, Georg Meinhardt, Jacob Katz, Joseph H. Kim, Pratyush K. Chaudhary, Chase Blagden, and Eric Xu. BigFinanceBench: A workflow-grounded benchmark for financial- research agents, 2026. URLhttps://arxiv.org/abs/2606.03829

    2026 arXiv

  2. [10]

    Hedge-Bench: Benchmarking agents on hard, realistic tasks pertaining to financial reasoning, 2026

    Eric Cho, Shawn Huang, Alice Lu, and Andy Lyu. Hedge-Bench: Benchmarking agents on hard, realistic tasks pertaining to financial reasoning, 2026. URLhttps://arxiv.org/abs/ 2606.03918

    2026 arXiv

  3. [11]

    FinResearchBench II: A deep research benchmark with consensus-derived gold rubrics for distinguishing financial report quality, 2026

    Beidi Luan, Rui Sun, Sinuo Wang, Yan Gu, Chao Li, Zhenliang Xiong, Jing Li, and Zuo Bai. FinResearchBench II: A deep research benchmark with consensus-derived gold rubrics for distinguishing financial report quality, 2026. URLhttps://arxiv.org/abs/2607.12252

    2026 arXiv

  4. [12]

    FinSearchComp: Towardsarealistic,expert-level evaluation of financial search and reasoning, 2025

    Liang Hu, Jianpeng Jiao, Jiashuo Liu, Yanle Ren, Zhoufutu Wen, Kaiyuan Zhang, Xuanliang Zhang, XiangGao, TianciHe, FeiHu, YaliLiao, ZaiyuanWang, ChenghaoYang, Qianyu Yang, Mingren Yin, Zhiyuan Zeng, Ge Zhang, Xinyi Zhang, Xiying Zhao, Zhenwei Zhu, Hongseok Namkoong,WenhaoHuang...

    2025

  5. [13]

    FinAgentBench: A benchmark dataset for agentic retrieval in financial question answering, 2025

    Chanyeol Choi, Jihoon Kwon, Alejandro Lopez-Lira, Chaewoon Kim, Minjae Kim, Juneha Hwang, Jaeseon Ha, Hojun Choi, Suyeol Yun, Yongjin Kim, and Yongjae Lee. FinAgentBench: A benchmark dataset for agentic retrieval in financial question answering, 2025. URLhttps: //arxiv.org/abs...

    2025

  6. [14]

    Fin-RATE: A real-world financial analytics and tracking evaluation benchmark for LLMs on SEC filings, 2026

    Yidong Jiang, Junrong Chen, Eftychia Makri, Jialin Chen, Peiwen Li, Ali Maatouk, Leandros Tassiulas, Eliot Brenner, Bing Xiang, and Rex Ying. Fin-RATE: A real-world financial analytics and tracking evaluation benchmark for LLMs on SEC filings, 2026. URLhttps: //arxiv.org/abs/2...

    2026 arXiv

  7. [15]

    FinGAIA: A chinese benchmark for AI agents in real-world financial domain, 2025

    Lingfeng Zeng, Fangqi Lou, Zixuan Wang, Jiajie Xu, Jinyi Niu, Mengping Li, Yifan Dong, Qi Qi, Wei Zhang, Ziwei Yang, Jun Han, Ruilun Feng, Ruiqi Hu, Lejie Zhang, Zhengbo Feng, Yicheng Ren, Xin Guo, Zhaowei Liu, Dongpo Cheng, Weige Cai, and Liwen Zhang. FinGAIA: A chinese bench...

    2025 arXiv

  8. [16]

    Pauli, Sarah Panda, Kidus Admassu, Said Bleik, Ademola Okerinde, and Jeremy Reynolds

    Wolfgang M. Pauli, Sarah Panda, Kidus Admassu, Said Bleik, Ademola Okerinde, and Jeremy Reynolds. FORCE-Bench: A benchmark, dataset, and evaluation harness for agentic AI in enterprise finance, 2026. URLhttps://arxiv.org/abs/2607.19409

    2026 arXiv

  9. [17]

    Smith, Jiahuan Pei, Arman Cohan, Jimin Huang, Yuehua Tang, Alejandro Lopez-Lira, Xi Chen, Xue Liu, Junichi Tsujii, Jian-Yun Nie, and Sophia Ananiadou

    Xueqing Peng, Zhuohan Xie, Yupeng Cao, Haohang Li, Lingfei Qian, Yan Wang, Vincent Jim Zhang, Huan He, Xuguang Ai, Linhai Ma, Ruoyu Xiang, Yueru He, Yi Han, Shuyao Wang, Yuqing Guo, Mingyang Jiang, Yilun Zhao, Youzhong Dong, Xiaoyu Wang, Yankai Chen, YeYuan,QiyuanZhang,FuyuanL...

    2026 arXiv

  10. [18]

    FinanceComplexQA: Benchmarking agentic reasoning on industrial-grade financial documents,

    Xianfu Cheng, Shiwei Zhang, Jiyu Zhao, Jian Yang, Xinyuan Wang, Ming Zhou, Weixiao Zhou, Xiangyuan Guan, Xiang Li, Zhenhe Wu, Ziyi Ni, Zhoujun Li, and Bingjing Xu. FinanceComplexQA: Benchmarking agentic reasoning on industrial-grade financial documents,

  11. [19]

    A critical evaluation of evaluations forlong-formquestionanswering

    Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. A critical evaluation of evaluations forlong-formquestionanswering. InProceedingsofthe61stAnnualMeetingoftheAssociation for Computational Linguistics (Volume 1: Long Papers), pages 3225–3245, Toronto, Canada, July 2023. A...

    2023 doi

  12. [20]

    G- Eval: NLG evaluation using GPT-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- Eval: NLG evaluation using GPT-4 with better human alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore, December 2023. A...

    2023 doi

  13. [21]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. InAdvances in Neural Information Processi...

    2023

  14. [22]

    Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero- Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: Evaluating large language models towards improved huma...

    2025 arXiv

  15. [23]

    CheckEval: A reliable LLM-as-a-judge framework for evaluating text generation using checklists

    Yukyung Lee, JoongHoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, and Najoung Kim. CheckEval: A reliable LLM-as-a-judge framework for evaluating text generation using checklists. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Proces...

    2025 doi

  16. [24]

    Hendryx, Brad Kenstler, and Bing Liu

    Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, Aishwarya Balwani, Denis Peskoff, Marcos Ayestaran, Sean M. Hendryx, Brad Kenstler, and Bing Liu. ResearchRubrics: A benchmark of pro...

    2025

  17. [25]

    ExpertLongBench: Benchmarking language models on expert-level long-form generation tasks with structured checklists, 2025

    Jie Ruan, Inderjeet Nair, Shuyang Cao, Amy Liu, Sheza Munir, Micah Pollens-Dempsey, Tiffany Chiang, Lucy Kates, Nicholas David, Sihan Chen, Ruxin Yang, Yuqian Yang, Jasmine Gump, Tessa Bialek, Vivek Sankaran, Margo Schlanger, and Lu Wang. ExpertLongBench: Benchmarking language...

    2025

  18. [26]

    URL https://samaya.ai/blog/criteria-eval

    ChristosBaziotis.Criteria-Eval: Evaluatinglong-formanswerstocomplexquestions.SamayaAI TechnicalBlog,June2025. URL https://samaya.ai/blog/criteria-eval. Contributors include Ashwin Paranjape, Jack Hessel, Jack Silva, and Mingyi Yang. Accessed 2026-07-31. 19

    2026

  19. [27]

    FinResearchBench: A logic tree based agent-as-a-judge evaluation framework for financial research agents

    Rui Sun, Zuo Bai, Wentao Zhang, Yuxiang Zhang, Li Zhao, Shan Sun, and Zhengwen Qiu. FinResearchBench: A logic tree based agent-as-a-judge evaluation framework for financial research agents. InProceedings of the 6th ACM International Conference on AI in Finance, pages 656–664. ...

    2025

  20. [28]

    FinDeepResearch: Evaluating deep research agents in rigorous financial analysis, 2025

    Fengbin Zhu, Xiang Yao Ng, Ziyang Liu, Chang Liu, Xianwei Zeng, Chao Wang, Tianhui Tan, Xuan Yao, Pengyang Shao, Min Xu, Zixuan Wang, Jing Wang, Xin Lin, Junfeng Li, Jingxian Zhu, Yang Zhang, Wenjie Wang, Fuli Feng, Richang Hong, Huanbo Luan, Ke-Wei Huang, and Tat-Seng Chua. F...

    2025

  21. [29]

    Deep FinResearch bench: Evaluating AI’sabilitytoconductprofessionalfinancialinvestmentresearch, 2026

    Mirazul Haque, Antony Papadimitriou, Samuel Mensah, Zhiqiang Ma, Zhijin Guo, Joy Prakash Sain, Simerjot Kaur, Charese Smiley, and Xiaomo Liu. Deep FinResearch bench: Evaluating AI’sabilitytoconductprofessionalfinancialinvestmentresearch, 2026. URL https://arxiv. org/abs/2604.21006

    2026 arXiv

  22. [30]

    MBABench: Evaluating LLM agents on end-to-end spreadsheet tasks in finance, 2026

    Thomson Yen, Julian Poeltl, Harshith Srinivas Gear, Yilin Meng, Joshua Fan, Adam Shen, Yili Liu, Ali Bauyrzhan, Siri Du, Haoyang Liu, Daniel Guetta, and Hongseok Namkoong. MBABench: Evaluating LLM agents on end-to-end spreadsheet tasks in finance, 2026. URL https://arxiv.org/a...

    2026 arXiv

  23. [31]

    FrontierFinance: A long-horizon computer-use benchmark of real-world financial tasks,

    Michael Krumdick, Varshini Reddy, Shivani Chaudhary, William Day, Maarij Ahmed, Hayan Haqqi, Muhammad Ahsen Fahim, Hanzallah Amjad, Ahmad Orakzai, Aqsa Gul, and Chris Tanner. FrontierFinance: A long-horizon computer-use benchmark of real-world financial tasks,

  24. [32]

    BrowseComp: A simple yet challenging benchmark for browsing agents, 2025

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp: A simple yet challenging benchmark for browsing agents, 2025. URLhttps://arxiv.org/abs/2504. 12516

    2025

  25. [33]

    GAIA: a benchmark for general ai assistants, 2023

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general ai assistants, 2023. URLhttps://arxiv.org/ abs/2311.12983

    2023 arXiv

  26. [34]

    BrowseComp-Plus: A more fair and transparent evaluation benchmark of deep-research agent, 2025

    Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. BrowseComp-Plus:...

    2025 arXiv

  27. [35]

    URLhttps://arxiv.org/abs/2604.05912

    arXiv

  28. [36]

    Jordan, Joseph E

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. InProceedings of the 41st ...

    2024

  29. [37]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023

    2023

  30. [38]

    NegativePoint

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952. doi: 10.2307/2334029. 21 A Sample queries and rubrics by use case The following six examples are the queries featured on t...

    1952 doi

  31. [39]

    DeepResearch Bench: A comprehensive benchmark for deep research agents, 2025

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. DeepResearch Bench: A comprehensive benchmark for deep research agents, 2025. URLhttps://arxiv. org/abs/2506.11763

    2025 arXiv

  32. [43]

    A financial query , which specifies the information the user is seeking

  33. [44]

    This is important for assessing the time understanding of the system

    The date the query was made . This is important for assessing the time understanding of the system . Whenever necessary , you should use this date as the temporal anchor for interpreting relative date terms in both the query and the rubrics

  34. [45]

    A financial report which aims to answer that query . 31

  35. [46]

    All of the input will be clearly marked in XML tags

    One or more natural language rubrics , each checking a specific aspect of the report . All of the input will be clearly marked in XML tags . Your task is to judge whether the report adequately satisfies each of the given rubrics . You must evaluate the report objectively and t...

  36. [47]

    Even in the case that one rubric seems related to another , you need to give your judgement of whether each rubric is satisfied independently

    ** Each rubric should be judged independently **. Even in the case that one rubric seems related to another , you need to give your judgement of whether each rubric is satisfied independently

  37. [48]

    USD 2.1 billion

    ** Pay attention to numerical units **. The report and the rubric might use different units to represent the same number . Take this into account when making your judgement . For example , "USD 2.1 billion " is equivalent to "USD 2 ,100 million "

  38. [49]

    3 ,098 million

    ** Accept reasonable numerical approximation **. A figure in the report is acceptable if it equals the rubric ’s figure after rounding the rubric ’s figure to the ( coarser ) precision the report uses . A figure stated at the same or finer precision than the rubric ’s, but wit...

  39. [2021]

    doi: 10.18653/v1/2021.acl-long.254

    Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.254. URL https://aclanthology.org/2021.acl-long.254/

    2021 doi

  40. [2025]

    doi: 10.18653/v1/2025.acl-long.766

    Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.766. URL https://aclanthology.org/2025.acl-long.766/

    2025 doi

  41. [2026]

    URLhttps://arxiv.org/abs/2607.19238

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.