REVIEW 4 major objections 6 minor 4 cited by
BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper introduces BizFinBench, a 6,781-question Chinese financial benchmark built from real investor queries, plus IteraJudge, an iterative LLM-judge method that tracks expert rankings more closely than vanilla LLM-as-a-judge; across…
desk verdict Useful Chinese financial benchmark built from real investor queries, but IteraJudge's validation is too thin to support the bias-reduction claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The dataset machinery is a workflow: collect real user queries from a financial QA platform, clean and classify them with GPT-4o, enrich each query with time-stamped market data and news, inject plausible distractors such as unrelated companies, opposite-sentiment articles, and temporally misplaced events, generate candidate answers with GPT-4o, then accept an item only after three senior financial experts independently agree on answer validity, contextual consistency, and category correctness. The evaluation machinery is IteraJudge's three-phase loop: sequential refinement $y_k = \mathrm{LLM}_{\mathrm{refine}}(y_{k-1} \parallel P(e_k, q))$ across dimensions $e_1,\dots,e_K$, a final refined answer $y_K$ used as an auto-generated quality benchmark, and a contrastive score $\mathrm{score}(y) = \mathrm{LLM}_{\mathrm{judge}}(q, y, y_K, E)$ that quantifies how far the original answer falls short of its refined version. The question-anchoring in every refinement step is what keeps the trajectory interpretable and comparable across models.
What would settle it
Re-annotate a random sample of the 6,781 items with an independent panel of experts who never see the original labels and compare the two panels' judgments; if agreement is low, or if re-generating the candidate answers with a different strong model such as DeepSeek-R1 and rerunning the unanimous-consensus process materially changes the ground truth, then the benchmark's reference signal is model-dependent and the reported rankings would need to be re-derived.
Extended reading notes
Core claim
The central claim is that realistic financial evaluation requires business-driven task design, and that BizFinBench supplies it: a 6,781-item Chinese-language benchmark whose questions originated from real investor queries, whose contexts include deliberately misleading distractors, and whose labels are fixed by unanimous expert consensus. A second claim is that LLM-based evaluation can be made less biased by iterative calibration: IteraJudge first rewrites the candidate answer through prompted refinements across preset evaluation dimensions, then has a judge score the original answer against the fully refined version; the gap between the two reveals dimensional deficiencies. The paper reports that IteraJudge improves agreement with expert rankings over vanilla LLM-as-a-judge by 3.09% to 17.24% on the Financial Data Description set and 4.44% to 11.37% on the Financial Tool Usage set, and that the resulting 25-model ranking shows no single winner, with strengths split across proprietary and open-source systems.
Load-bearing premise
The load-bearing premise is that unanimous agreement among three senior financial experts on answers drafted by GPT-4o, sitting inside contexts that GPT-4o helped construct, yields unbiased ground truth for the benchmark; if those labels are systematically tilted by the generator's blind spots, every model ranking and the IteraJudge calibration inherits the same tilt.
Editorial extensions
If this is right
- Model rankings on BizFinBench become a reusable reference point for anyone choosing a model for Chinese-language finance work: no single model tops all nine task categories, and proprietary models take the top three spots on eight of nine.
- Numerical calculation and reasoning reward scale: within the Qwen3 family scores rise consistently from the 1.7B to the 32B model, while the smallest model trails by dozens of points.
- Temporal reasoning is the sharpest separator, with GPT-4o beating Llama-3.1-8B-Instruct by 32.19 points, so it should be a focus for improvement.
- Emotion recognition and stock-price prediction remain hard for every evaluated model, so high-stakes deployment in those areas needs extra safeguards.
- IteraJudge, used as the scoring layer, produces Spearman correlations with expert rankings that are up to 17.24% higher than vanilla LLM-as-a-judge on the two tested datasets.
Reading between the lines
- Extension: because the reference answer in IteraJudge is itself generated by a prompted LLM before expert validation, a natural follow-up is to test whether the same gains survive when the reference generator and the judge model are decoupled, for instance by judging with one model while references come from another.
- Extension: the dimension-wise refinement trajectory that IteraJudge produces could be reused as a diagnostic signal in other precision domains such as legal or medical QA; the paper does not explore this transfer.
- Extension: the modest absolute Spearman values on FDD, roughly 0.31 to 0.57, suggest IteraJudge improves ordinal agreement but is not yet a substitute for expert panels; a practical user should read IteraJudge scores as a more reliable signal rather than as ground truth.
- Extension: releasing the benchmark enables a direct external audit: re-run the 25-model evaluation with a different judge model and check whether the ranking pattern, especially the size of IteraJudge gains, is stable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces BizFinBench, a Chinese-language financial benchmark of 6,781 real-user queries from the iwencai platform, organized into five dimensions and nine task categories, with gold labels produced by three senior financial experts who validate GPT-4o-generated candidate answers. The authors also propose IteraJudge, an iterative LLM-based evaluation method that refines a model output along disentangled dimensions and scores the original answer by contrast with the refined version. They benchmark 25 proprietary and open-source LLMs, reporting accuracy and judge-based scores per task, and validate IteraJudge on two tasks (FDD and FTU) by comparing Spearman correlations with expert ratings against vanilla LLM-as-a-Judge.
Significance. If the claims hold, the benchmark would be a useful business-grounded addition to the financial LLM evaluation suite, with a realistic Chinese-language query distribution and expert-validated labels, plus a reusable judge framework. Strengths I credit: the dataset is sourced from a real financial assistant platform rather than exam questions; the three-expert unanimous-consensus annotation protocol is a strong quality signal; the authors benchmark 25 models with a documented setup and disclose computational costs; the ablation compares IteraJudge against a vanilla LLM-as-a-Judge baseline across four different judge models, which is a reasonable first check; and the limitations appendix is candid about answer-extraction errors and coverage gaps. The headline contribution of IteraJudge as a bias-reduction method is, however, supported by only partial evidence (two of nine tasks, no sample sizes, no significance tests), and the evaluation pipeline shares a model family across data construction, refinement, and judging, so the bias-reduction claim needs more evidence before it can be taken at face value.
major comments (4)
- [§4.4, Table 4] The central claim that IteraJudge 'reduces bias when LLMs serve as evaluators' rests on Spearman correlations computed on only two of the nine tasks (FDD and FTU), with no sample sizes, no confidence intervals, and no significance tests reported. The reader cannot tell whether the observed gains (e.g., 0.4848 to 0.5684 for GPT-4o on FDD) are statistically distinguishable from noise, or whether the same improvements hold on the seven other tasks. I ask the authors to report the number of expert-rated responses, provide confidence intervals or a permutation test for each Spearman difference, and ideally validate on at least the judge-scored tasks (FQA, FTU) and one accuracy task, such as FNC or FTR.
- [§3.1 and §4.4, Eqs. (1)-(2)] There is a self-referential loop between the data-generation process and the judge validation: GPT-4o is used in Section 3.1 to clean, classify, synthesize, and generate candidate answers that three experts then edit or confirm, and the same class of model (GPT-4o) is also the default judge and the refinement engine inside IteraJudge. Since the reference yK in Eq. (1) is produced by the same model family that is later scored, improved agreement with expert labels could reflect self-consistency with GPT-4o's refinement style rather than genuine bias reduction. This does not invalidate the benchmark, but it does mean the IteraJudge validation needs to break the loop. A concrete test: hold out a set of expert-written or expert-revised reference answers (rather than GPT-4o-refined ones), or run the judge with the reference signal removed, and report whether the Spearman gains persist. Without such a test, the abstract's claim about bias reduction is overstated.
- [§3.1, data construction; Table 1] The paper states that every data point is accepted only after three experts reach 'full consensus across all aspects', yet no statistics about the annotation process are given: how many candidate answers were rejected in the first pass, how many iterations were needed, and what the inter-annotator agreement was before consensus. Without these numbers, the reader cannot assess whether the unanimous consensus reflects a clean dataset or simply a process that iterates until someone yields. Please report the initial agreement rate, the number of rounds, and the rejection rate, at least for a random subsample.
- [§4.3, Table 3] The main benchmark results for the two judge-scored tasks (FQA and FDD, and to some extent FTU) are produced by GPT-4o as the unified judge, but the prompt and rubric for these judge scores are only shown as figures in Appendix D, and no evidence of judge reliability (e.g., agreement with the expert labels) is provided for the full nine-task set. Since IteraJudge is only validated on two tasks with a single evaluated model (Qwen2.5-7B-Instruct), the Table 3 scores for FQA, FDD, and FTU rest on the unvalidated assumption that GPT-4o's judge scores correlate with expert judgment on the full benchmark. Please either validate the judge on all tasks or caveat the affected columns and averages.
minor comments (6)
- [Abstract and §1] The abstract and introduction call BizFinBench 'the first benchmark specifically designed to evaluate LLMs in real-world financial applications'; earlier financial-domain benchmarks also use industry data (for example FinQA and FLARE include financial filings and prediction tasks), so 'first' is a strong claim that should be softened or explicitly scoped to business-driven user queries from a financial assistant platform.
- [§3.2, Table 2] Table 2 reports 6,781 total instances, but Table 1 says 7,016 and Appendix Table 6 counts sum to 6,786; the discrepancy between the three numbers should be reconciled.
- [§3.1] The text says '10 distinct task types' and 'nine representative task categories' in the same subsection, then Table 2 lists nine categories; please align the wording.
- [Table 3] The table uses golden/silver/bronze highlighting, but a color-blind reader may not distinguish the shades; consider adding numeric markers or a 'top-3' column.
- [Appendix B, Limitation (1)] The limitation statement that rule-based answer extraction 'may introduce an estimated 2% error' is plausible but no basis for the 2% is given; specify how the estimate was derived.
- [Appendix D] The evaluation prompts in Figures 5-7 are screenshots; please provide the exact text in the appendix or supplementary material so that the judge setup is reproducible.
Circularity Check
No significant circularity: BizFinBench's expert-validated labels and IteraJudge's external Spearman comparison break the self-reference chain.
full rationale
Although GPT-4o is used both for data construction (cleaning, classification, synthesis, and initial candidate answers in Section 3.1) and as the judge/refiner within IteraJudge (Section 4.2, Eqs. 1-2), the paper's central claims do not reduce to self-definition. The benchmark's ground truth is not GPT-4o's own output: Section 3.1 states that candidate answers 'are not directly included in the dataset' and that 'every entry is independently reviewed and annotated by three senior financial experts' with 'full consensus' required for acceptance. Thus model rankings on BizFinBench are measured against expert-approved labels, not against the generator. Similarly, IteraJudge's claimed bias reduction is validated empirically against expert evaluation: Section 4.4 reports Spearman correlations on FDD and FTU, comparing vanilla LLM-as-a-Judge and IteraJudge, which is an external criterion rather than a fitted or definitionally forced outcome. The 'auto-generated quality benchmark' yK in Eq. (1) is a model-internal reference, which creates a legitimate independence concern and a statistical weakness (no confidence intervals or sample sizes are reported), but it does not make the derivation circular because the method's success is not defined as agreement with yK; it is measured by agreement with human experts. The only self-citations (e.g., Lu et al. [2024] and Zhang et al. [2023a]) are background references, not load-bearing. Overall, the derivation chain is self-contained with external validation, so no circular step can be exhibited.
Assumptions & free parameters
assumptions (4)
- domain assumption Real user queries from the iwencai platform are representative of real-world financial business tasks.
- domain assumption Unanimous agreement among three senior financial experts yields correct ground-truth answers.
- domain assumption GPT-4o-based cleaning, classification, and synthesis does not introduce systematic bias into categories, distractors, or answers.
- domain assumption Spearman correlation with expert ratings on FDD and FTU is sufficient evidence that IteraJudge reduces LLM-judge bias.
Cite this review
Pith. "Pith review of BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs." pith.science (2026). https://pith.science/paper/Q7JHQDWR
@misc{pith2026250519457,
author = {Pith},
title = {Pith review of: BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/Q7JHQDWR}},
note = {Machine review of arXiv:2505.19457}
}
read the original abstract
Large language models excel in general tasks, yet assessing their reliability in logic-heavy, precision-critical domains like finance, law, and healthcare remains challenging. To address this, we introduce BizFinBench, the first benchmark specifically designed to evaluate LLMs in real-world financial applications. BizFinBench consists of 6,781 well-annotated queries in Chinese, spanning five dimensions: numerical calculation, reasoning, information extraction, prediction recognition, and knowledge-based question answering, grouped into nine fine-grained categories. The benchmark includes both objective and subjective metrics. We also introduce IteraJudge, a novel LLM evaluation method that reduces bias when LLMs serve as evaluators in objective metrics. We benchmark 25 models, including both proprietary and open-source systems. Extensive experiments show that no model dominates across all tasks. Our evaluation reveals distinct capability patterns: (1) In Numerical Calculation, Claude-3.5-Sonnet (63.18) and DeepSeek-R1 (64.04) lead, while smaller models like Qwen2.5-VL-3B (15.92) lag significantly; (2) In Reasoning, proprietary models dominate (ChatGPT-o3: 83.58, Gemini-2.0-Flash: 81.15), with open-source models trailing by up to 19.49 points; (3) In Information Extraction, the performance spread is the largest, with DeepSeek-R1 scoring 71.46, while Qwen3-1.7B scores 11.23; (4) In Prediction Recognition, performance variance is minimal, with top models scoring between 39.16 and 50.00. We find that while current LLMs handle routine finance queries competently, they struggle with complex scenarios requiring cross-concept reasoning. BizFinBench offers a rigorous, business-aligned benchmark for future research. The code and dataset are available at https://github.com/HiThink-Research/BizFinBench.
Figures
Figures from the paper (14 more)
Forward citations
Cited by 4 Pith papers
-
Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?
C-SUITEBENCH shows that adding visual business evidence improves evidence-centric reasoning in nine multimodal LLMs but degrades constrained resource allocation in all nine, a pattern attributed to signal crowding.
-
Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
A new benchmark shows LLMs' financial calculation accuracy collapses without explicit formulas and degrades further when they must generate multi-metric tables.
-
StockSim: A Dual-Mode Order-Level Simulator for Evaluating Multi-Agent LLMs in Financial Markets
StockSim provides a dual-mode simulated stock market, with order-level and candlestick-level execution, for evaluating LLM trading agents.
-
FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality
A multi-LLM consensus pipeline turns 14,450 auto-generated candidate rubrics into 2,600 distinguishable gold rubrics that rank 10 financial deep-research systems from 58.58% to 22.23% pass rate.
Reference graph
Works this paper leans on
-
[1]
投资收益:如果你现在有 100 元,你可以选择将它存入银行或投资于其他项目,从而在未来获得更多的钱。因此,今天的 100 元比未来的 100 元更有价值,因为你有机会通过投资增加它的价值。
-
[2]
通货膨胀:随着时间的推移,由于通货膨胀的影响,货币的购买力会逐渐下降。这意味着,同样数量的货币在未来可能无法购买 到与现在相同数量的商品和服务。
-
[3]
风险与不确定性:未来是不确定的,承诺在未来支付的款项可能因为各种原因(如违约、经济衰退等)而无法兑现。因此,人们 通常更偏好当前的确定性收益而非未来的不确定性收益。 在金融学中,货币时间价值通常通过现值( Present Value, PV )和终值( Future Value, FV )的概念来量化。现值是指未来某一时点 的一定金额按照一定的利率折算到现在的价值;终值则是指现在的一笔资金按照一定的利率增长到未来某一时点的价值。通过这些计算,可以帮助个人和企业做出更合理的财务决策。 Figure 10: An example instance from the Financial Knowledge QA dataset. 18 根据下面的提供的数据描述这支股票的情况,包括变化趋势和一些数值 : 根据提供...
work page 2025
-
[4]
**2025 年 1 月 11 日至 1 月 14 日 ** :股票继续下跌 1.48% ,但随后在 1 月 14 日反弹 3.45% ,显示出市场情绪有所恢复。3
**2025 年 1 月 10 日 ** :股票大幅下跌 9.97% ,这可能是由于市场对该公司或行业出现了负面消息,导致投资者信心下降。2. **2025 年 1 月 11 日至 1 月 14 日 ** :股票继续下跌 1.48% ,但随后在 1 月 14 日反弹 3.45% ,显示出市场情绪有所恢复。3. **2025 年 1 月 15 日至 1 月 16 日 ** :股票再次下跌 2.75% ,但 1 月 16 日没有变化,市场进入观望状态。
work page 2025
-
[5]
**2025 年 1 月 17 日至 1 月 21 日 ** :股票连续小幅上涨,涨幅分别为 1.19% 、 1.33% 和 2.62% ,显示出市场信心逐渐增强。5. **2025 年 1 月 22 日至 1 月 24 日 ** :股票再次出现波动, 1 月 22 日下跌 2.12% , 1 月 23 日大幅下跌 6.51% , 1 月 24 日小幅下跌 1.55% 。
work page 2025
-
[6]
**2025 年 1 月 27 日 ** :股票大幅上涨 10.06% ,显示出市场情绪的强烈反弹。7. **2025 年 2 月 5 日至 2 月 6 日 ** :股票再次大幅下跌 10.00% ,随后在 2 月 6 日继续下跌 6.19% ,市场情绪再次转为悲观。 ### 数值分析- **2025 年 2 月 6 日的收盘价 ** : 5.91 元- **2025 年 2 月 6 日的开盘价 ** : 5.67 元 - **2025 年 2 月 6 日的最高价 ** : 6.04 元- **2025 年 2 月 6 日的最低价 ** : 5.67 元 ### 总体趋势- ** 波动性 ** :在这段时间内,北京文化的股票价格波动较大,多次出现大幅涨跌,显示出市场对该股票的不确定性较高。 - ** 市场...
work page 2025
-
[7]
完整提取:所有出现的实体都需要记录,不进行去重,即使某个实体多次出现,也需要完整罗列。
-
[8]
逐条列出:按照文本顺序依次提取实体,确保输出顺序与原文匹配。
Show all 39 references
-
[9]
results": [ {
请以纯文本形式输出 JSON ,不要包含任何代码块标记(如 ```json 或 ``` ),参考格式如下: { "results": [ { "type": "xxx", "text": "xxx" }, { "type": "xxx", "text": "xxx" }, { "type": "xxx", "text": "xxx" } ] } Financial Named Entity Recognition { "results": [ { "type": "Organization", "text": " 东方电热 " }, { "type...
2023
-
[10]
投资者看到某家科技公司发布了一项突破性技术,认为这将推动股价大幅上 涨。
-
[11]
市场经济数据连续几个季度显示强劲增长,投资者预期未来几年股市将持续 上扬。
-
[12]
新兴市场国家推出了一系列经济改革政策,投资者对该地区的长期发展前景 感到乐观。
-
[13]
某行业正处于快速增长阶段,投资者相信自己的投资组合中相关资产会显著 增值。
-
[14]
投资者参加了一场金融论坛,听到多位专家对未来的正面预测后更加坚信市 场潜力。 二、焦虑 - 特点:对不确定性感到不安,担心潜在风险或损失。 - 示例场景:
-
[15]
投资者持有的股票价格短期内波动剧烈,导致其频繁查看账户以确认资产状 况。
-
[16]
国际政治局势紧张,投资者担心战争或制裁可能引发全球经济衰退。
-
[17]
央行宣布即将加息,投资者担忧利率上升会对债券和房地产投资造成负面影 响。
-
[18]
某个关键行业的龙头企业突然宣布业绩下滑,投资者对其相关资产的前景感 到忧虑。
-
[19]
市场传言某国可能会实施资本管制,投资者担心资金无法顺利撤回。 三、消极 - 特点:对市场或投资结果产生失望、沮丧或悲观的情绪。 - 示例场景:
-
[20]
投资者长期持有的一只股票因公司丑闻而暴跌,导致其亏损严重。
-
[21]
全球经济进入衰退周期,投资者看到自己的投资组合价值持续缩水。
-
[22]
某行业受到政策监管加强的影响,投资者对该行业的未来失去信心。
-
[23]
投资者尝试了多种策略但均未取得理想收益,开始怀疑自己的投资能力。
-
[24]
市场长期处于低迷状态,投资者对任何新的投资机会都提不起兴趣。 四、兴奋 - 特点:因市场上涨或投资成功而感到愉悦和满足。 - 示例场景:
-
[25]
投资者买入的一只股票因公司财报超预期而涨停,当天收益达到两位数。
-
[26]
某热门板块突然成为市场焦点,投资者持有的相关资产迅速升值。
-
[27]
投资者通过精准判断抓住了一次短期交易机会,获得超额回报。
-
[28]
市场因利好消息出现普涨行情,投资者发现自己几乎所有资产都在增值。
-
[29]
投资者参与的新股申购中签,并在上市首日获得了高额收益。 五、冷静 - 特点:保持中立和理性,不受市场波动或情绪干扰。 - 示例场景:
-
[30]
市场因突发新闻出现短暂下跌,但投资者并未恐慌抛售,而是选择继续观 察。
-
[31]
投资者在市场剧烈波动时坚持既定的投资计划,不轻易调整策略。
-
[32]
面对市场热点板块的快速上涨,投资者并未盲目追高,而是等待更好的入场 时机。
-
[33]
投资者在经济数据不佳时仍保持信心,认为这只是短期现象而非趋势性变 化。
-
[34]
即使市场出现重大事件,投资者也能基于数据分析做出客观判断,避免情绪 化决策。 六、后悔 - 特点:对过去的决策感到遗憾或自责,尤其是当结果不如预期时。 - 示例场景:
-
[35]
投资者因犹豫不决错过了某只股票的最佳买入时机,事后发现该股票大幅上 涨。
-
[36]
投资者在市场高位时卖出资产,随后市场继续上涨,导致错失更多收益。
-
[37]
投资者未能及时止损,导致原本的小幅亏损变成重大损失。
-
[38]
投资者听信他人建议进行了一次高风险投资,最终失败并造成巨大损失。
-
[39]
cot": "< 请在此处填写你的推理过程或判断依据,描述清楚得出结论的理由 >
投资者在市场下跌时被迫平仓,事后发现市场很快反弹,导致错失恢复机 会。 请严格按照以下 JSON 格式输出你的分析结果: { "cot": "< 请在此处填写你的推理过程或判断依据,描述清楚得出结论的理由 >", "result": ["< 请填写你的情绪分类结果 1>","< 请填写你的情绪分类 结果 2>"] } [ 背景信息和用户问句 ] {" 市场环境 ": { " 大盘走势分析 ": {" 走势表现 ": { " 三大指数表现 ": " 上证指数微涨 0.3% ,深证成指上涨 0.5% ,创业板指上涨 0.7% ,市场呈现温和回升态势 "...
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.