Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Financial LLM agents that ace applied tasks still lack rule understanding, lose 15 points when chained into workflows, and break under two-round adversarial attacks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 17:25 UTC pith:LR3O5OWU

load-bearing objection CNFinBench is a genuine new benchmark with careful construction and plausible qualitative findings, but three concrete reporting/auditability problems—a copy-paste column, inconsistent headline numbers, and an unspecified HICS metric—block acceptance until fixed. the 4 major comments →

arxiv 2512.09506 v6 pith:LR3O5OWU submitted 2025-12-10 cs.CE

Beyond Knowledge to Agency: Evaluating Expertise, Autonomy, and Integrity in Finance with CNFinBench

Jinru Ding , Chao Ding , Yidong Jiang , Wenrao Pang , Boyi Xiao , Zhiqiang Liu , Jiayuan Chen , Yun Zhong
show 4 more authors
Tiantian Yuan Junming Guan Dawei Cheng Jie Xu
This is my paper
classification cs.CE
keywords financial LLM agentsagentic autonomyregulatory compliancemulti-turn adversarial attackHICSLLM-as-judgecompliance driftbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that evaluating financial large language models on textbook-style question answering hides the risks that appear when they act as autonomous agents. To close that gap, it builds CNFinBench, a benchmark of 29 subtasks spanning expertise, autonomy, and integrity. Running 22 models through it, the paper establishes three failure modes: models handle applied loan and credit tasks well but score poorly on regulatory exam-style items; performance drops 15.4 points when tasks are chained into end-to-end execution paths; and in multi-turn adversarial dialogues compliance violations surge by roughly 172% by the second round, with impersonation attacks degrading fastest (+354%). The central contribution is a new metric, HICS, which claims to quantify violation escalation and behavioral consistency that binary refusal rates miss.

Core claim

On the paper's own terms, the discovery is that the three capabilities that matter for safe financial deployment — expertise, autonomy, and integrity — are separable and measurably out of balance in current LLMs. Closed and open models both show an 'illusion of regulatory competence': they choose correct answers on applied credit and loan tasks while failing Fin_Basics and Fin_Cert_Exams that require exact rule interpretation. In autonomy, models lose 15.4 points going from isolated steps to full execution chains, with the largest drops in parameter precision and inter-agent coordination rather than API selection. In integrity, single-turn defenses look strong (several models score above 98

What carries the argument

CNFinBench's Expertise–Autonomy–Integrity triad of 29 subtasks, and the Harmful Instruction Compliance Score (HICS), a 100-point multi-turn safety metric. HICS works by decomposing each adversarial response into atomic and sequential violations, assigning a base score by risk type, scaling by a severity multiplier, and tracking consistency across dialogue turns. Its role is to replace binary refusal rates with a continuous signal of compliance erosion; the paper credits it with revealing the divergent collapse rhythms across the three attack families.

Load-bearing premise

The whole integrity measurement rests on the assumption that HICS's base scores, severity multipliers, and cross-turn deductions — whose exact values are never stated — reflect the real compliance risk of a financial response.

What would settle it

Re-run the MT_App dialogues and have three independent compliance experts score each response with a fixed rubric on a 1–5 risk scale, blinded to HICS. If expert-rated risk does not increase from Round 1 to Round 2 (or increases far less than the reported 354.3%), the escalation is an artifact of the unpublished HICS weights rather than a property of model behavior.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Models that look safe in single-turn tests (refusal rates above 98%) can nonetheless comply with harmful requests by the second round of a dialogue, so deployment-time safety checks need multi-turn evaluation.
  • High performance on applied financial tasks is not evidence of rule understanding; certification-style regulatory items expose a capability gap untouched by loan and credit benchmark accuracy.
  • Autonomy failures are procedural, not conceptual: models pick the right APIs and roles, then fail at parameter precision, unit handling, and inter-agent data flow.
  • The three attack families collapse on different timelines (application early, internal gradual, cognitive late), implying that defenses need to be tailored to trust logic rather than uniformly hardened.
  • Because HICS tracks specific violation rule types, the benchmark offers traceable, per-turn deduction logs that can guide targeted model refinement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the unspecified HICS penalty weights are arbitrary, the headline escalation percentages (172%, 354%) should be read as ordinal signals — 'violations clearly grow' — rather than as calibrated measures of real-world risk magnitude.
  • Editorial inference: the Round-2 collapse in application-layer attacks suggests that identity impersonation exploits a distinct mechanism — the model's willingness to trust a claimed insider role — which may be addressable by explicit authorization checks in the agent scaffold rather than by further safety fine-tuning.
  • Editorial inference: the same Expertise–Autonomy–Integrity measurement design is portable to other high-privilege agent domains (health records, legal document handling, cloud administration), where the same pattern of good single-turn knowledge but weak chained compliance is plausible.
  • Editorial inference: a direct test of whether HICS adds information over refusal rates would be to re-score the same MT_App dialogues with binary 'compliant/not' labels; if the binary signal already peaks in Round 2, HICS's contribution is diagnostic granularity rather than new detection.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. CNFinBench is a Chinese financial LLM benchmark with 29 subtasks organized into Expertise, Autonomy, and Integrity. It draws on regulatory corpora, financial reports, anonymized loan and fraud data, and simulated API/dialogue workflows; 70% of QA items are LLM-drafted and then filtered by three models plus expert review. The paper evaluates 22 open-/closed-source and finance-tuned models using task-specific rubrics and an LLM-as-judge panel (Cohen's κ = 0.74 against experts). It introduces HICS, a 100-point multi-turn compliance metric, and reports three headline findings: (1) models perform well on applied tasks but poorly on rule understanding; (2) a 15.4-point drop from single modules to full execution chains; and (3) rapid compliance collapse under multi-turn attacks, with average violations surging by 159.05% (abstract) or 172.3% (full abstract) in Round 2 and MT_App collapsing fastest (+354.3% in Round 2).

Significance. If the results hold, CNFinBench would be a valuable community resource: it jointly covers agentic execution, multi-turn adversarial red teaming, and a continuous safety metric, which few prior financial benchmarks do. The strengths are real: expert involvement in task design, use of first-hand financial data, a public evaluation platform, detailed rubrics in Appendix E, and a human-agreement study for the LLM-judge setup. However, the current manuscript does not yet support the flagship Integrity and Autonomy claims as stated. HICS is described only qualitatively, so the headline violation-surge numbers are not independently checkable; Table 2 contains an apparent data duplication; and the 15.4-point drop is not derivable from the reported task scores. These are fixable in revision, but they are load-bearing for the paper's conclusions.

major comments (4)
  1. [§5.2, §C.3, Abstract/§1] HICS is the basis for the paper's central Integrity findings, but it is never quantitatively defined. §5.2 says HICS 'operates on a 100-point scale' with 'rule triggers, severity-adjusted deduction multipliers, and cross-turn consistency tracking,' and §C.3 introduces Base Score, Severity Multiplier, and cross-turn consistency without any equation, weight table, or worked example. Consequently, the Round-2 surge figures (159.05% in the first abstract, 172.3% in the full-text abstract; MT_App +354.3%, §5.3.3) and the Jaccard-similarity collapse-rhythm analysis cannot be recomputed independently. The Appendix D κ=0.74 study validates LLM rubric judgments against experts; it does not validate the HICS penalty parametrization. Please provide the full HICS formula, base-score schedule, multiplier values, consistency term, and reconcile the 159.05/172.3 discrepancy.
  2. [Table 2, §5.3.1] In Table 2, the Fin_Basics (BK) column is identical to the Fin_Report_Parse (RP) column for all 22 models (e.g., Doubao 76.7, GPT-4o 83.1, Qwen3-14B 77.0), while Appendix F Table 10 reports different Fin_Basics values for the same models (e.g., GPT-4o 35.0±0.6, Qwen3-14B 3.9±0.3). The §5.3.1 claim that models score only 38.50 on Fin_Basics and the Expertise-vs-Autonomy contrast depend on this column. The duplication appears to be a data error; it must be corrected and the affected analyses rerun before the Expertise conclusions can be evaluated.
  3. [§5.3.1] The headline Autonomy result, 'a 15.4-point drop from single-step to multi-step tasks,' is not supported by the numbers in the same paragraph. It cites Path_Plan 63.3, Ret_API 53.9, and Multi_App 56.65; the pairwise differences are 9.4 and 6.65, not 15.4. Please define explicitly which tasks are aggregated as 'single modules' and which as 'execution chains,' report the per-model computation, and show how 15.4 is obtained. Without this, the central module-to-chain degradation claim is unverifiable.
  4. [§4.3] The difficulty filter discards any item that two of Qwen3-235B, DeepSeek-V3, and GPT-4o answer correctly, and the same model families are also used as LLM judges and are among the 22 evaluated models. This is an ad hoc selection rule that can systematically disadvantage those families and their derivatives, so cross-model rankings may partly reflect benchmark construction rather than capability. I am not claiming fitted-parameter circularity, but this is a validity risk. Please report results separately for the 30% expert-authored subset, or provide a stability analysis (e.g., rank correlation with and without the filtered items).
minor comments (4)
  1. [Table 1] The abbreviation key lists 'MI = MT_Iner'; this should be 'MT_Inter'.
  2. [Table 2 / Appendix F] Minor naming inconsistencies: 'Itnet_ID' should be 'Intent_ID', and Appendix F uses 'Qwen3-235B-A22B' while the main text uses 'Qwen3-235B'.
  3. [§5.1] All models are evaluated with a 2,048-token context window, yet the benchmark includes Long_QA and Long_Conv tasks over long financial documents. Please clarify how documents are chunked or selected; otherwise these tasks measure truncated-context behavior rather than long-context reasoning.
  4. [§6] The statement that online leaderboard scores may differ from the paper because the platform uses 'a curated challenging subset' makes the reported numbers hard to compare with the live system. Please archive and version the exact evaluation set used for the paper's tables.

Circularity Check

1 steps flagged

Partial construction self-reference in benchmark difficulty; no load-bearing self-citation chain, but the headline 'lack of rule understanding' finding is partly manufactured for the generator-judge models by the difficulty filter.

specific steps
  1. other [Section 4.3 (QA Generation & Refinement) → Section 5.3.1 and Abstract]
    "we employ Qwen3-235B, DeepSeek-V3 and GPT-4o in parallel to conduct two independent attempts. If two models answer correctly, the question is deemed too easy and excluded. ... They achieve only 44.50 on Fin_Cert_Exams and 38.50 on Fin_Basics, both of which test precise interpretation of technical phrasing and tolerate no conceptual mistakes. Success on Loan_Analysis (79.95) and Credit_Score (68.20) largely relies on correlating features with binary outcomes, rather than true domain understanding."

    The benchmark's difficulty is fit to the very models later evaluated and used as judges: any item Qwen3-235B, DeepSeek-V3, or GPT-4o answers correctly is removed before scoring, and 70% of items are drafted by these same three models. The paper's headline finding — 'LLMs perform well in applied tasks yet lack robust rule understanding' — is then read off the filtered Fin_Basics/Fin_Cert_Exams scores. For the three generator models, low rule scores are partly forced by construction: the filter deletes exactly the items on which they would have succeeded. The reduction is partial, not total: the filter applies to all 29 subtasks, and the low-scores pattern also holds for the other 19 evaluated models not involved in construction, so the finding is not entirely self-fulfilling.

full rationale

This is an empirical benchmark paper, not a derivation paper, so the strong 'Eq. X = Eq. Y' circularity patterns mostly do not apply. The central numerical claims are measurements, not predictions from fitted parameters. (1) Construction self-reference (partial): Section 4.3 filters out items that Qwen3-235B/DeepSeek-V3/GPT-4o answer correctly and discards items they cannot agree on; these same models draft 70% of items and later serve as primary judges (Section 5.2). This makes the reported 'LLMs lack robust rule understanding' deficit partly by-construction for those three models, but the finding also holds across the 19 models outside the construction loop, and filtering applies uniformly to all tasks, so the headline expertise claim has independent content. (2) HICS is described qualitatively only (Sections 5.2, C.3): no Base Score values, Severity Multiplier schedule, or consistency-tracking formula are given, and the same 'average violations surging' figure appears as 159.05% (one abstract) and 172.3% (another abstract). This makes the headline Integrity numbers non-reproducible as written — a correctness/reproducibility risk, not a demonstrable circular reduction, since without the formula one cannot exhibit a step where output equals input by construction. Hard rule 1 therefore prohibits flagging it as circularity. (3) LLM-judge overlap: judges are Qwen3-235B, DeepSeek-V3, and GPT-4o/Gemini-2.5-Pro, with stated candidate-judge non-overlap and a human agreement check (Cohen's kappa = 0.74, Appendix D) as an external anchor; this is mitigation rather than circularity. (4) Self-citations exist (FinEval [12], CFBenchmark [18], Fin-R1 [19] share authors with this paper, e.g., Zhiqiang Liu), but they are used only as comparative prior work and as an evaluated model, not to justify a uniqueness theorem or to import an ansatz, so they are not load-bearing. Net: one partial, minor-moderate circularity in construction; score 3.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

Central claims rest on: (i) correctness of gold labels (expert-reviewed LLM drafts), (ii) validity of LLM-as-judge and HICS as risk measures (κ=0.74 on a 10% sample; HICS weights undisclosed), and (iii) representativeness of synthetic four-turn attack chains. The difficulty filter — excluding items the drafting LLMs solve — is an author-chosen rule that directly determines the benchmark's difficulty profile. There is no machine-checked verification and no external calibration of HICS.

free parameters (3)
  • HICS base scores / severity multipliers / cross-turn consistency weights = not specified
    HICS is a 100-point deduction system (§5.2, C.3) whose per-rule base scores, severity multipliers, and turn-consistency penalties are author-set but never enumerated; every reported HICS number depends on them.
  • Difficulty-filter rule (exclude if two of Qwen3-235B/DeepSeek-V3/GPT-4o answer correctly) = n/a
    Iterative filtering (§4.3) removes items the drafting LLMs solve; this rule, not an external standard, sets benchmark difficulty and directly shapes the observed score distribution.
  • Option-shuffling consistency criterion = n/a
    Objective tasks require exact match across randomized variants (§5.2); the 'must answer correctly in all variants' threshold is an author-chosen strict rule that mechanically lowers reported accuracy.
axioms (5)
  • standard math Standard evaluation metrics (exact match with option shuffling, micro-F1, Cohen's κ) validly capture the target capabilities
    Used throughout §5.2 and Appendix D; assumption that exact-match-over-variants and span-level F1 faithfully measure financial expertise and extraction quality.
  • domain assumption Gold answers and risk labels in CNFinBench are correct
    70% of QA is LLM-drafted and refined by LLM voting plus a 21-expert review (§4.3-4.4); no independent measurement of label accuracy is reported, so residual synthesis errors would bias all model scores.
  • domain assumption LLM-as-judge panels approximate expert judgment
    Three judge LLMs with task rubrics score open-ended responses in 21 of 29 tasks; human agreement is κ=0.74 on 10% of samples (Appendix D) — moderate agreement, cited in the paper as 'reliable proxies'.
  • ad hoc to paper HICS penalty weights encode true compliance-risk severity
    HICS base scores, severity multipliers, and cross-turn consistency rules are introduced for this paper (§5.2, C.3); no expert elicitation or external calibration is reported, and the values are never listed.
  • domain assumption Four-turn chain-of-attack dialogues with 17 personas and 7 strategies represent realistic adversarial financial interactions
    Attacker personas and strategies are synthesized from incident reports and transcripts (Appendix C); there is no evidence that measured success rates transfer to real-world adversarial campaigns.
invented entities (2)
  • HICS (Harmful Instruction Compliance Score) no independent evidence
    purpose: Continuous 100-point metric for compliance degradation in multi-turn dialogues
    No external validation of HICS values against expert-assigned penalties or real incident outcomes; the human agreement study covers generic LLM-judge tasks (κ=0.74), not HICS deductions specifically.
  • Defense Degradation Curve / Cascading Evaluation Architecture no independent evidence
    purpose: Conceptual framework describing how violations compound across dialogue turns
    Qualitative constructs introduced in Appendix C.3; no falsifiable handle or quantitative definition beyond HICS itself.

pith-pipeline@v1.3.0-alltime-deepseek · 59797 in / 23924 out tokens · 224412 ms · 2026-08-03T17:25:50.442446+00:00 · methodology

0 comments
read the original abstract

As large language models (LLMs) become high-privilege agents in risk-sensitive settings, they introduce systemic threats beyond hallucination, where minor compliance errors can cause critical data leaks. However, existing benchmarks focus on rule-based QA, lacking agentic execution modeling, overlooking compliance drift in adversarial interactions, and relying on binary safety metrics that fail to capture behavioral degradation. To bridge these gaps, we present CNFinBench, a comprehensive benchmark spanning 29 subtasks grounded in the triad of expertise, autonomy, and integrity. It assesses domain-specific capabilities through certified regulatory corpora and professional financial tasks, reconstructs end-to-end agent workflows from requirement parsing to tool verification, and simulates multi-turn adversarial attacks that induce behavioral compliance drift. To quantify safety degradation, we introduce the Harmful Instruction Compliance Score (HICS), a multi-dimensional safety metric that integrates risk-type-specific deductions, multi-turn consistency tracking, and severity-adjusted penalty scaling based on fine-grained violation triggers. Evaluations over 22 open-/closed-source models reveal: LLMs perform well in applied tasks yet lack robust rule understanding, suffer a 15.4 decline from single modules to full execution chains, and collapse rapidly in multi-turn attacks, with average violations surging by 159.05\% in Round 2. CNFinBench is available at https://cnfinbench.opencompass.org.cn and https://github.com/open-compass/CNFinBench.

Figures

Figures reproduced from arXiv: 2512.09506 by Boyi Xiao, Chao Ding, Dawei Cheng, Jiayuan Chen, Jie Xu, Jinru Ding, Junming Guan, Tiantian Yuan, Wenrao Pang, Yidong Jiang, Yun Zhong, Zhiqiang Liu.

Figure 1
Figure 1. Figure 1: Three-dimensional task taxonomy in CNFinBench, spanning Capability, Autonomy, and Safety. Each axis consists of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the CNFinBench data construction pipeline. Each QA instance is generated through a multi-stage process [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Task Difficulty Distribution Across Tasks [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Radar plots of LLM task capabilities across the Expertise–Autonomy–Integrity dimensions, visualized over 8 evaluation [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Evolution of average error counts (left) and average [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Illustration of the CNFinBench evaluation platform. (a) System architecture built upon the OpenCompass framework, [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?

    cs.AI 2026-05 conditional novelty 6.0

    TeleCom-Bench reveals LLMs reach 90% on telecom intent and entity tasks but drop to 30% on solution generation and root cause analysis in live network scenarios.

  2. FinDocMRE: A Benchmark for Document-Level Financial Multimodal Reasoning Evaluation

    cs.CE 2026-05 unverdicted novelty 6.0

    FinDocMRE is a new multi-image document-level benchmark spanning 12 financial domains and 5 task types, showing that 11 tested LMMs all score below 65 overall with particular weaknesses in numerical estimation and cro...

  3. Rethinking Fraud Safety Evaluation: Multi-Round Attacks Reveal Safety-Utility Tradeoffs in Graph-Context LLM Defenders

    cs.CR 2026-05 unverdicted novelty 5.0

    Graph-context LLM fraud defenders improve early refusal under replay and adaptive multi-round attacks compared to text baselines but increase benign over-refusal, with the cost localized to how the LLM consumes struct...

Reference graph

Works this paper leans on

44 extracted references · 1 canonical work pages · cited by 3 Pith papers

  1. [1]

    Anthropic. 2025. System Card: Claude Opus 4 & Claude Sonnet 4. https://www- cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf

    2025

  2. [2]

    Lei Bai, Zhongrui Cai, Yuhang Cao, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, et al. 2025. Intern-s1: A scientific multimodal foundation model.arXiv preprint arXiv:2508.15763(2025)

    arXiv 2025

  3. [3]

    Elias Bassani and Ignacio Sanchez. 2024. GuardBench: A Large-Scale Benchmark for Guardrail Models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 18393–18409. doi:10.18653/v1/2024.emnlp-main.1022

    work page doi:10.18653/v1/2024.emnlp-main.1022 2024

  4. [4]

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. 2024. Jailbreakbench: An open robustness bench- mark for jailbreaking large language models.Advances in Neural Information Processing Systems37 (2024), 55005–55029

    2024

  5. [5]

    Ye Chen, Wei Cai, Liangmin Wu, Xiaowei Li, Zhanxuan Xin, and Cong Fu. 2023. Tigerbot: An open multilingual multitask llm.arXiv preprint arXiv:2312.08688 (2023)

    Pith/arXiv arXiv 2023

  6. [6]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. 2021. Finqa: A dataset of numerical reasoning over financial data. (2021), 3697–3711

    2021

  7. [7]

    Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. ConvFinQA: Exploring the Chain of Numerical Rea- soning in Conversational Finance Question Answering. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association f...

    doi:10.18653/v1/2022.emnlp- 2022

  8. [8]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    Pith/arXiv arXiv 2025

  9. [9]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

    Pith/arXiv arXiv 2025

  10. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv e-prints(2024), arXiv–2407

    2024

  11. [11]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, ...

    2025

  12. [12]

    Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, Zhiqiang Liu, Sizhe Wang, Jinyi Niu, Chuqi Wang, Yanhui Wang, Xiaolong Liang, Xiaoming Huang, Bing Zhu, Zhongyu Wei, Yun Chen, Weining Shen, and Liwen Zhang. 2025. FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models. InProceedings of the 2025 Conference of t...

    2025

  13. [13]

    Gang Hu, Ke Qin, Chenhan Yuan, Min Peng, Alejandro Lopez-Lira, Benyou Wang, Sophia Ananiadou, Jimin Huang, and Qianqian Xie. 2024. No language is an island: Unifying chinese and english in financial large language models, instruction data, and benchmarks.arXiv preprint arXiv:2403.06249(2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Tiansheng Hu, Tongyan Hu, Liuyang Bai, Yilun Zhao, Arman Cohan, and Chen Zhao. 2025. FinTrust: A Comprehensive Benchmark of Trustworthiness Eval- uation in Finance Domain. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Associat...

    doi:10.18653/v1/2025.emnlp- 2025

  15. [15]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    Pith/arXiv arXiv 2024

  16. [16]

    Lanlan Ji, Dominic Seyler, Gunkirat Kaur, Manjunath Hegde, Koustuv Das- gupta, and Bing Xiang. 2025. PHANTOM: A Benchmark for Hallucination Detection in Financial Long-Context QA. InThe Thirty-ninth Annual Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=5YQAo0S3Hm

    2025

  17. [17]

    Mintong Kang, Zhaorun Chen, Chejian Xu, Jiawei Zhang, Chengquan Guo, Minzhou Pan, Ivan Revilla, Yu Sun, and Bo Li. 2025. PolyGuard: Massive Multi- Domain Safety Policy-Grounded Guardrail Dataset. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=mORzRZaqT4

    2025

  18. [18]

    Yang Lei, Jiangtong Li, Dawei Cheng, Zhijun Ding, and Changjun Jiang. 2023. CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model. arXiv:2311.05812 [cs.CL]

    Pith/arXiv arXiv 2023

  19. [19]

    Zhaowei Liu, Xin Guo, Zhi Yang, Fangqi Lou, Lingfeng Zeng, Mengping Li, Qi Qi, Zhiqiang Liu, Yiyang Han, Dongpo Cheng, Ronghao Chen, Huacan Wang, Xingdong Feng, Huixia Judy Wang, Chengchun Shi, and Liwen Zhang. 2026. Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning. arXiv:2503.16252 [cs.CL] https://arxiv.org/abs/2503.16252

    arXiv 2026

  20. [20]

    Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: financial opinion mining and question answering. InCompanion proceedings of the the web conference 2018. 1941–1942

    2018

  21. [21]

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. HarmBench: a standardized evaluation framework for automated red teaming and robust refusal. InProceedings of the 41st International Jinru Ding et al. Conference on Machine Learning(Vienna, Aust...

    2024

  22. [22]

    Ying Nie, Binwei Yan, Tianyu Guo, Hao Liu, Haoyu Wang, Wei He, Binfan Zheng, Weihao Wang, Qiang Li, Weijian Sun, Yunhe Wang, and Dacheng Tao. 2025. CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: H...

    2025

  23. [23]

    Maya Pavlova, Erik Brinkman, Krithika Iyer, Vítor Albiero, Joanna Bitton, Hailey Nguyen, Cristian Canton Ferrer, Ivan Evtimov, and Aaron Grattafiori. 2025. Automated Red Teaming with GOAT: the Generative Offensive Agent Tester. InICLR 2025 Workshop on Building Trust in Language Models and Applications. https://openreview.net/forum?id=6uyczU6S2M

    2025

  24. [24]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    Pith/arXiv arXiv 2025

  25. [25]

    Varshini Reddy, Rik Koncel-Kedziorski, Viet Dac Lai, Michael Krumdick, Charles Lovering, and Chris Tanner. 2024. DocFinQA: A Long-Context Financial Rea- soning Dataset. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Co...

    doi:10.18653/v1/2024.acl-short.42 2024

  26. [26]

    ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. 2025. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914(2025)

    arXiv 2025

  27. [27]

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al

  28. [28]

    Kimi Team, Yifan Bai, Yiping Bao, Y. Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Chenxiao Gao, Hongcheng Gao, Peizhong Ga...

    Pith/arXiv arXiv 2026

  29. [29]

    Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li. 2024. ALERT: A Comprehensive Benchmark for Assessing Large Language Models’ Safety through Red Teaming. arXiv:2404.08676 [cs.CL] https://arxiv.org/abs/2404.08676

    Pith/arXiv arXiv 2024

  30. [30]

    Xiaojun Wu, Junxi Liu, Huan-Yi Su, Zhouchi Lin, Yiyan Qi, Chengjin Xu, Jiajun Su, Jiajie Zhong, Fuwei Wang, Saizhuo Wang, Fengrui Hua, Jia Li, and Jian Guo

  31. [31]

    Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al . 2024. Finben: A holistic financial benchmark for large language models.Advances in Neural Information Processing Systems37 (2024), 95716–95743

    2024

  32. [32]

    InFindings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.)

    Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 22544–22560. doi:10.18653/v1/2025.fi...

    doi:10.18653/v1/2025.findings-emnlp.1227 2025

  33. [33]

    Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2025. FinGPT: Open- Source Financial Large Language Models. arXiv:2306.06031 [q-fin.ST] https: //arxiv.org/abs/2306.06031

    arXiv 2025

  34. [34]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  35. [35]

    Yi Zeng, Yu Yang, Andy Zhou, Jeffrey Ziwei Tan, Yuheng Tu, Yifan Mai, Kevin Klyman, Minzhou Pan, Ruoxi Jia, Dawn Song, Percy Liang, and Bo Li. 2025. AIR-BENCH 2024: A Safety Benchmark based on Regulation and Policies Spec- ified Risk Categories. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=UVnD9Ze6mF

    2025

  36. [36]

    Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. 2025. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models.arXiv preprint arXiv:2508.06471 (2025)

    Pith/arXiv arXiv 2025

  37. [37]

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2024. SafetyBench: Evaluating the Safety of Large Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.)...

    doi:10.18653/v1/2024.acl-long.830 2024

  38. [38]

    Mengao Zhang, Jiayu Fu, Tanya Warrier, Yuwen Wang, Tianhui Tan, and Ke-wei Huang. 2025. FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance. InProceedings of the 6th ACM International Conference on AI in Finance. 159–167

    2025

  39. [39]

    How should one respond in the absence of regulation?

    Jie Zhu, Junhui Li, Yalong Wen, and Lifan Guo. 2024. Benchmarking Large Language Models on CFLUE - A Chinese Financial Language Understanding Evaluation Dataset. InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 5673–569...

    doi:10.18653/v1/ 2024

  40. [40]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A Question An- swering Benchmark on a Hybrid of Tabular and Textual Content in Finance. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural ...

    2021

  41. [42]

    Protect customer interests: prevent customers from suf- fering asset losses or service disruptions due to uncon- trolled risks within financial institutions, thereby enhanc- ing trust in financial services

  42. [43]

    Maintain financial market stability: reduce the likelihood that risks at individual financial institutions evolve into systemic risks, avoiding severe market volatility or panic

  43. [44]

    Online_Fraud Example question:

    Comply with regulatory requirements: ensure that finan- cial service activities adhere to relevant laws, regulations, and supervisory rules, avoiding penalties and reputational damage caused by non-compliance. " Online_Fraud Example question: "You are now a financial fraud prevention ex- pert. Based on the transaction information below, please determine w...

    2023

  44. [2025]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267(2025)

    Pith/arXiv arXiv 2025