{"id":"ee70102a-8b60-4633-a741-5e33decb9d16","arxiv_id":"2501.06211","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"This paper introduces a Chinese financial LLM benchmark with certification and scenario tasks and reports that Baichuan4-Finance outperforms five other models on most tasks.","lead":"FLAME is a Chinese-language benchmark for financial large language models, built from 14 professional certification exams and about 100 business application scenarios. The paper's comparison of six models suggests Baichuan4-Finance leads on most tasks, but the benchmark's data and scoring details are not fully public.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contamination check absent: Baichuan4-Finance's near-ceiling FLAME-Cer scores may reflect memorized public exam questions, undermining the headline ranking.","rationale":"The paper's central claim has two parts: FLAME is a comprehensive Chinese financial evaluation system, and Baichuan4-Finance excels other tested models on it. The second part is the headline empirical result. The most load-bearing assumption for that result is that FLAME-Cer scores reflect financial competence rather than memorization. The certification questions are drawn from public, widely available exam banks; several case-study examples (e.g., the CFA ethics vignette and the CPA auditing question) are recognizable practice questions. Baichuan4-Finance is a finance-tuned model whose training data very plausibly includes such materials. Without a contamination check, the reported 94% average and individual subject scores above 97% are consistent with answer memorization rather than generalization. This is not an accusation of fraud; it is a standard validity threat for any benchmark built from public exams, and the paper provides no evidence against it. The FLAME-Sce manual scoring also lacks inter-rater reliability, but that affects a smaller portion of the evaluation and is secondary to the memorization threat for the flagship ranking. The reader's weakest_assumption listed contamination alongside annotator credentials and inter-rater agreement; my focus on contamination is a partial agreement, because I consider it the single decisive issue. The proposed paraphrase test is concrete and would settle whether the ranking survives when surface form is changed. Without such evidence, the conditional verdict is appropriate: the benchmark idea is plausible, but the headline claim is unsupported as stated. If the paraphrase test shows stable performance, the conditional can be lifted. If it shows a large drop for Baichuan4-Finance, the ranking should be reinterpreted or rejected.","tokens_in":10524,"tokens_out":3242,"duration_ms":34452,"concrete_test":"Test for memorization via paraphrase robustness: select a random subset of FLAME-Cer questions (e.g., 200 per certification); create paraphrased versions that preserve the underlying financial problem but alter surface features (numbers, institution names, wording, option order). Evaluate Baichuan4-Finance, Qwen2.5-72B, and GPT-4o under the same zero-shot protocol on both original and paraphrased sets. If Baichuan4-Finance's accuracy drops materially (e.g., >10 points) on paraphrases while general models degrade less or by a similar margin, the original scores are inflated by memorization. For a stronger check, use only questions published after Baichuan4-Finance's training cutoff, if any; stable performance on those would refute contamination. Release the benchmark questions and splits so the check is externally reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FLAME-Cer is built from 14 authoritative certification exams (CPA, CFA, FRM, FundPQ, SPQ, etc.) whose questions are publicly available in practice banks. The paper reports Baichuan4-Finance at 93.62% average accuracy, with FundPQ 97.93% and SPQ 97.60%, while GPT-4o scores 78.23%. Such near-ceiling performance on a 16,000-question benchmark is exactly the signature of training-set overlap. Baichuan4-Finance is a finance-specialized model likely pretrained or fine-tuned on large financial corpora including exam questions. Section 2.1 only states questions were 'manually reviewed to ensure accuracy'; there is no decontamination analysis, no temporal split, no provenance check against pretraining data, and no release of the evaluation subset or splits. If the certification questions (or near-duplicates) appeared in Baichuan4-Finance's training data, FLAME-Cer measures memorization for that model, and the central claim that it 'excels other LLMs in most tasks' is not supported. The absence of a dataset release makes this impossible for reviewers to check independently.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FLAME, a Chinese-language evaluation system for financial LLMs, comprising FLAME-Cer (approximately 16,000 multiple-choice questions from 14 financial certification exams) and FLAME-Sce (about 5,000 scenario-based questions across 10 primary and roughly 100 tertiary financial tasks). Six models—GPT-4o, ERNIE-4.0-Turbo-128K, GLM-4-PLUS, Qwen2.5-72B-Instruct, XuanYuan3-70B-Chat, and Baichuan4-Finance—are evaluated zero-shot. The authors report that Baichuan4-Finance achieves the highest average accuracy (93.62%) on FLAME-Cer and the highest usability rate (84.15%) on FLAME-Sce, concluding that it excels in most tasks.","tokens_in":10694,"tokens_out":2053,"duration_ms":20201,"significance":"If the benchmark is validated, FLAME would be a useful contribution: it is one of the first Chinese-language financial evaluation suites covering both certification knowledge and practical business scenarios, and it targets a gap left by English-centric benchmarks such as FinBen and FinanceBench. The two-part design (knowledge plus application) is sensible, and the public GitHub repository is a positive step for reproducibility. However, the paper's central empirical claim—that Baichuan4-Finance is the best among the tested models—is currently supported only by point estimates from a dataset whose contamination status, annotation reliability, and statistical uncertainty are not established. The benchmark's value therefore depends on fixing these validation gaps.","major_comments":[{"comment":"The paper provides no contamination analysis. FLAME-Cer draws from public certification exams whose questions appear in freely accessible practice banks, and Baichuan4-Finance achieves near-ceiling scores on FundPQ (97.93%) and SPQ (97.60%). Without a decontamination check (e.g., n-gram overlap with pretraining corpora, temporal splits, or release of evaluation subsets) the headline ranking in Table 1 could reflect memorization rather than financial competence. This is load-bearing for the central claim and must be addressed before the results can be interpreted.","section":"Section 2.1 and Table 1"},{"comment":"FLAME-Sce scores are assigned by human evaluators on a -1 to 5 scale, but the paper reports neither the number of annotators, their qualifications, nor inter-rater agreement (e.g., Cohen's kappa or Krippendorff's alpha). The final usability rates in Table 2 are therefore presented without any measure of scoring reliability. Since the scenario-level averages drive the comparison between models, this omission is a major threat to the validity of the FLAME-Sce results.","section":"Section 2.4 and Table 2"},{"comment":"The scenario-specific dimension weights (e.g., Accuracy 30%, Completeness 30%, Clarity 10%) and the usability thresholds (>=3 for simple, >=4 for complex) are choices made by the authors without external validation or sensitivity analysis. Because the final scores are linear weighted sums, a different but equally plausible weight configuration could change which model ranks first in several scenarios. The paper should include a sensitivity analysis over weights and thresholds, or justify these choices with reference to domain expert consensus.","section":"Section 2.4.2-2.4.3 and Table 2"},{"comment":"All reported results are point estimates with no confidence intervals, standard errors, or significance tests. For example, GPT-4o (78.23%) and ERNIE-4.0-Turbo (77.03%) on FLAME-Cer are reported as different, but with roughly 16,000 questions the difference may be statistically significant; conversely, smaller differences on individual certification subjects (e.g., GLM-4-PLUS 81.45% vs Qwen2.5 82.92% on CFA) may be within sampling noise. The paper should report binomial confidence intervals or perform paired significance tests (e.g., McNemar's test) for the pairwise model comparisons.","section":"Tables 1-2 and Section 3.2-3.3"}],"minor_comments":[{"comment":"The certification name is misspelled as 'Ecomonist' in the table; it should be 'Economist'.","section":"Table 1"},{"comment":"The text 'financial risk control (8587%)' appears to be a typo; it should read '85-87%'.","section":"Section 3.3"},{"comment":"The capitalization of 'Flame-Cer' in Figure 1 is inconsistent with 'FLAME-Cer' used elsewhere in the paper.","section":"Section 2.1 and Figure 1"},{"comment":"The case studies are presented as anecdotal illustrations without a systematic selection criterion; adding a note that these are examples rather than evidence would help avoid over-interpretation.","section":"Section 3.4"},{"comment":"For FLAME-Cer, the metric is 'accuracy' but the paper does not specify whether the benchmark uses only single-answer multiple choice or also multi-answer questions; the Economist case study shows a multi-answer item, so the scoring rule for partial credit should be stated explicitly.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable benchmark contribution, but its central ranking claim is not yet supported due to the absence of contamination checks, inter-rater reliability, and statistical inference. These are fixable with additional analysis and reporting. I would also encourage the editor to treat the GitHub repository's contents as part of the review; if the dataset is not actually released, the benchmark cannot be independently validated. The paper's fit for cs.CL is fine, but it would benefit from a stronger evaluation methodology section."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FLAME is worth taking seriously as a resource. The certification benchmark is broader than anything comparable for Chinese finance — 14 exams, about 16,000 questions, with a sensible three-level significance split — and the scenario taxonomy (10 primary, 21 secondary, ~100 tasks) is a real attempt to measure application ability rather than just text overlap. The tables are internally consistent, the zero-shot setup is clear, and the case examples give a concrete sense of what is being tested. The citations to PIXIU, FinBen, FinanceBench, and BizBench are appropriate. That is real work, and it is a contribution.\n\nThe soft spot is exactly where the stress-test note points. Baichuan4-Finance scores 93.62% average on FLAME-Cer, with 97.93% and 97.60% on FundPQ and SPQ, both of which are public certification exams. That near-ceiling performance is the signature of training-set overlap, and the paper does not report any contamination check, temporal split, or provenance analysis. Without that, the central claim that Baichuan4-Finance 'excels other LLMs in most tasks' is not supported; it may simply be memorizing questions that are already in its training data. The problem is compounded by the fact that the dataset is not released, so reviewers cannot check.\n\nThere are smaller issues. FLAME-Sce scores come from human rubric scoring, but there is no inter-rater agreement, no annotator credentials, and no confidence intervals or sample sizes for any of the headline numbers. The scenario weights and usability thresholds in Section 2.4 are defensible but arbitrary, and they are not externally validated. None of these are fatal on their own; together they mean the paper currently reads as a benchmark announcement rather than a validated evaluation study.\n\nThe paper is coherent and the authors clearly know the domain. I would send it to a serious referee, but with the dataset release and contamination analysis treated as gatekeeping issues. If the data stays private and the contamination question is not addressed, the ranking should not be published as a finding. If those are fixed, this is a useful addition to the Chinese financial LLM evaluation landscape.","headline":"FLAME is a genuinely useful Chinese financial LLM benchmark resource, but the paper's headline ranking is not yet supported because the near-ceiling certification scores are not checked for training-data contamination.","tokens_in":11251,"tokens_out":2401,"would_cite":false,"duration_ms":24067,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FLAME, a new Chinese-language financial LLM evaluation system, spans 14 certifications and nearly 100 business scenarios, and on both benchmarks Baichuan4-Finance outperforms GPT-4o, Qwen2.5, GLM-4, ERNIE-4.0, and XuanYuan3.","keywords":["financial LLM evaluation","benchmark","Chinese financial NLP","certification exams","financial business scenarios","Baichuan4-Finance","zero-shot evaluation"],"falsifier":"Run the same six models on a freshly written set of certification questions that postdate the models' training cutoffs (or perform a membership-inference test on FLAME-Cer items); if Baichuan4-Finance's lead shrinks or disappears, the ranking reflects memorization of public exam questions rather than financial competence.","tokens_in":10283,"feed_emoji":"📊","tokens_out":2962,"duration_ms":27279,"temperature":0.7,"pith_summary":"This paper introduces FLAME, a Chinese-language evaluation system for financial large language models, and uses it to compare six models. The system has two parts: FLAME-Cer, a set of about 16,000 manually reviewed questions from 14 financial certification exams, and FLAME-Sce, over 5,000 scenario-based tasks covering almost 100 real-world financial business applications. The central claim is that FLAME provides a professional, comprehensive standard for judging financial LLMs, and that when measured against this standard, Baichuan4-Finance leads the tested models on most tasks. A sympathetic reader would care because existing financial benchmarks mostly score from an NLP perspective, whereas FLAME claims to evaluate actual financial competence and practical deployment readiness.","feed_headline":"New Chinese finance benchmark crowns Baichuan4-Finance","feed_subtitle":"FLAME tests 14 certifications and ~100 business tasks; the finance-tuned model beats GPT-4o and Qwen2.5.","key_machinery":"The load-bearing object is the two-benchmark system. FLAME-Cer uses accuracy on manually reviewed certification questions as its metric, with questions drawn from 14 authoritative financial certifications (CPA, CFA, FRM, and others) and grouped into three significance levels. FLAME-Sce uses a hierarchical structure of 10 primary, 21 secondary, and nearly 100 tertiary financial business scenarios, scored by human raters on eight basic dimensions (accuracy, completeness, compliance, relevance, professionalism, clarity, practicality, and instruction compliance) with scenario-specific weights; scores range from -1 to 5, a weighted final score is computed, and usability thresholds differ by scenario complexity.","core_discovery":"The paper claims that FLAME is a comprehensive financial LLM evaluation system in Chinese, consisting of FLAME-Cer and FLAME-Sce, and that evaluating six representative models reveals Baichuan4-Finance excels other LLMs in most tasks. On FLAME-Cer, Baichuan4-Finance achieves an average accuracy of 93.62%, ahead of Qwen2.5-72B-Instruct (88.24%), GLM-4-PLUS (81.17%), GPT-4o (78.23%), ERNIE-4.0-Turbo-128K (77.03%), and XuanYuan3-70B-Chat (72.07%). On FLAME-Sce, Baichuan4-Finance leads with a usability rate of 84.15%, followed by GPT-4o (79.88%), Qwen2.5 (79.18%), ERNIE-4.0 (78.01%), GLM-4 (76.92%), and XuanYuan3 (61.34%). The paper also observes that the China Actuarial Association exam is the hardest for all models (best score 66.51%), while models generally exceed 80% on fund, securities, and banking qualification exams, and that all models score lower in financial analysis and document generation scenarios.","pith_inferences":["Because FLAME-Cer questions come from public certification exams, the ranking could partly reflect memorization if any test items appeared in model training data; the paper reports no contamination check.","The manual scoring scheme of FLAME-Sce might be reproducible by an LLM judge, which would allow cheaper, broader evaluation, but would need validation against the human scores.","A natural extension is to create parallel benchmarks in other languages, since the scenario taxonomy is not inherently Chinese-specific.","The reported usability thresholds suggest that a financial institution could treat a model as deployable in a scenario only if it crosses the relevant threshold; whether those thresholds match actual regulatory or business requirements is not tested."],"forward_implications":["If FLAME works as claimed, it gives financial institutions a standardized Chinese-language way to compare LLMs on both professional knowledge and practical business tasks.","Baichuan4-Finance is currently the best of the six tested models on this benchmark, especially on certification accuracy and scenario usability.","The hardest certification area for all tested models is actuarial science, suggesting a concrete gap in current financial LLM knowledge.","The weak performance of all models in financial analysis and document generation points to common deficiencies that future financial LLMs need to address.","FLAME-Sce's thresholds for 'usable' versus 'not usable' responses offer a deployment-oriented criterion beyond raw accuracy."],"supporting_citations":[{"why":"PIXIU is cited as an existing financial LLM benchmark that evaluates from an NLP perspective, the gap FLAME aims to fill.","marker":"[6]"},{"why":"FinBen is cited as a holistic financial benchmark that FLAME contrasts with, motivating the need for a professional financial perspective.","marker":"[5]"},{"why":"FinanceBench is cited as a financial question-answering benchmark that represents the prior focus on text similarity and n-gram overlap.","marker":"[3]"},{"why":"BizBench is cited as a quantitative reasoning benchmark for business and finance that FLAME positions against.","marker":"[4]"}],"fun_headline_variants":["FLAME benchmark crowns Baichuan4-Finance over GPT-4o","Chinese finance LLM test: Baichuan4-Finance wins top spot","Baichuan4-Finance outshines GPT-4o on 14 financial exams","New FLAME benchmark: finance-tuned model beats all rivals","FLAME eval: Baichuan4-Finance leads on certs and tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that the benchmark's ground-truth answers and human scenario scores are accurate, and that the questions have not appeared in model training data.","fun_headline_variants_meta":{"raw":{"variants":["FLAME benchmark crowns Baichuan4-Finance over GPT-4o","Chinese finance LLM test: Baichuan4-Finance wins top spot","Baichuan4-Finance outshines GPT-4o on 14 financial exams","New FLAME benchmark: finance-tuned model beats all rivals","FLAME eval: Baichuan4-Finance leads on certs and tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000319,"raw_usage":{"total_tokens":1861,"prompt_tokens":1064,"completion_tokens":797,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":691}},"tokens_in":680,"tokens_out":797,"duration_ms":7070,"temperature":1.0,"reasoning_tokens":691,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:21:25.196742+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same six models on a freshly written set of certification questions that postdate the models' training cutoffs (or perform a membership-inference test on FLAME-Cer items); if Baichuan4-Finance's lead shrinks or disappears, the ranking reflects memorization of public exam questions rather than financial competence.","supporting_citations":[],"review_version":1}