REVIEW 4 major objections 5 minor 12 references
FEVO: Financial Knowledge Expansion and Reasoning Evolution for Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read FEVO-R32B, a 32-billion-parameter model trained with continued pre-training, supervised fine-tuning, and reinforcement learning, is claimed to achieve state-of-the-art accuracy on five Chinese financial benchmarks, beating much larger…
desk verdict A transparent, well-built financial LLM pipeline whose SOTA claim is not interpretable: the 215-item Fin CPA evaluation set matches the RL training set in name and size, so the headline numbers are likely memorization until held-out evaluation and decontamination are provided. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the sequential three-stage training recipe. First, CPT expands the model's financial knowledge by training on about 188M tokens of financial textbooks, exam questions, and general data. Second, SFT instills a structured reasoning pattern—plan, reasoning, reflection, backtracking, answer—by fine-tuning on 291K chain-of-thought traces distilled from a frontier reasoning model. Third, RL integrates the two using a modified version of the VAPO (Value-Augmented Proximal Policy Optimization) algorithm with rule-based rewards, balanced batching, and a language-consistency penalty. The designed object that carries the argument is the FEVO-Train dataset suite: each stage has its own filtered dataset, and the RL set is made of open-ended questions (converted from multiple-choice questions) to prevent guessing and reward hacking.
What would settle it
Evaluate FEVO-R32B and FEVO-R32B-0 on a newly assembled set of Chinese financial exam questions that were written after the model's training data cutoff and never used in any stage of the FEVO pipeline, including the MCQ-to-open-ended conversion step. If the accuracy gap between R32B and R32B-0 collapses, or if R32B's scores fall below the state-of-the-art levels reported on the original benchmarks, the central claim that the pipeline teaches transferable financial reasoning would be falsified.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that expanding a 32B model's financial knowledge before teaching it structured reasoning, and then integrating the two by reinforcement learning, yields a model that outperforms much larger and finance-specialized competitors on Chinese financial and taxation benchmarks. Specifically, FEVO-R32B is reported to reach state-of-the-art accuracy on Fin CPA, Fin CCR, FinanceIQ, CFLUE, and FIN-EVA, and to markedly outperform FEVO-R32B-0, the same base model trained with only RL. The paper interprets the gap between the two models as direct evidence that the CPT and SFT stages contribute real capability beyond what RL alone induces.
Load-bearing premise
The reported superiority of FEVO-R32B over its RL-only control and over other models assumes that the evaluation benchmarks were not used during training; the in-house 215-item Fin CPA set appears in both the RL training data (Appendix A.1) and the evaluation set (Appendix A.2), and CFLUE and FinanceIQ derive from the same open sources used for training.
Editorial extensions
If this is right
- If the reported results hold, a 32B open-weight model can match or exceed much larger closed models and finance-specialist systems on Chinese financial exams, lowering the cost of deploying domain-expert-level financial reasoning.
- The comparison against the RL-only control (FEVO-R32B-0) implies that continued pre-training and structured SFT contribute substantial capability gains on top of RL, not just training noise.
- Converting multiple-choice questions to open-ended ones is presented as a way to curb reward hacking and expand the size of usable RL training data.
- Balanced batching in the RL stage is reported to speed up training by about 20% without degrading performance, making the pipeline more practical.
Reading between the lines
- The reported SOTA numbers on Fin CPA, CFLUE, and FinanceIQ may partly reflect benchmark leakage: the 215-item Fin CPA set is used in both RL training and evaluation, and CFLUE/FinanceIQ come from the same open sources used for training; a truly held-out evaluation set would be needed to confirm transfer.
- A direct test of the framework's generality would be to apply the same CPT-SFT-RL recipe to another knowledge-heavy domain (e.g., law or medicine) and see if the pattern of gains over an RL-only control repeats.
- The claim that the gap between FEVO-R32B and FEVO-R32B-0 comes from knowledge expansion and structured reasoning could be further probed by ablating one stage at a time (CPT only + RL, or SFT only + RL) on a contamination-free benchmark.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FEVO, a three-stage post-training framework (CPT, SFT, RL) built on Qwen2.5-32B for financial-domain tasks, and curates stage-specific datasets FEVO-Train-C/S/R. The authors report that the resulting FEVO-R32B achieves state-of-the-art accuracy on five financial benchmarks compared to larger general models and financial specialists, and that its margin over FEVO-R32B-0 validates the contribution of the CPT and SFT stages. The evaluation evidence, however, is compromised by direct overlap between training and evaluation data: the 215-item in-house Fin CPA set appears both in RL training (Appendix A.1) and in the evaluation benchmark (Appendix A.2), and the CFLUE and FinanceIQ benchmarks overlap in source or content with training corpora. The paper reports no decontamination checks, so the headline results and the R32B-versus-R32B-0 gap may be memorization artifacts rather than measures of generalization.
Significance. If the reported results were valid, FEVO would be a meaningful contribution: the pipeline is clearly described, the ablations across C32B/S32B/R32B and R32B-0 are well designed, and the engineering details (balanced batching, reward shaping, open-ended conversion for RL) are useful for practitioners. The central claim, however, is that benchmark scores measure generalization, and that condition is not met in the current manuscript. The direct train/evaluation overlap on Fin CPA and the plausible overlap through CFLUE and FinanceIQ invalidate the state-of-the-art claim and the validation of the framework's components. Without a decontaminated evaluation on held-out data, the experimental evidence cannot support the abstract's conclusions.
major comments (4)
- [Appendix A.1/A.2, Tables 2 and 3] The in-house Fin CPA set is listed as RL training data with 215 items in Table 2 and as an evaluation benchmark with 215 items in Table 3, with no statement that the two sets are disjoint. This means the model is trained on the exact questions on which it is evaluated. Consequently, the Fin CPA column in Table 1 (FEVO-R32B 73.60 vs. FEVO-R32B-0 63.04) cannot be interpreted as evidence of generalization, and the claim of state-of-the-art performance on Fin CPA is unsupported.
- [Section 2.2.3 and Appendix A.1/A.2] The RL data include "Dianjin Fill-in-blanks" described as "Extracted from CFLUE (no overlap with SFT data)", while the SFT data include 20K CFLUE items and the evaluation uses the CFLUE benchmark with 3,864 items. The parenthetical only asserts disjointness between two training stages; it does not rule out overlap with the evaluation set. Since the paper reports no n-gram, MinHash, or embedding decontamination against the evaluation benchmarks, the CFLUE results in Table 1 are not a valid measure of out-of-training generalization.
- [Section 2.1 and Appendix A.2] FinanceIQ (Duxiaoman) is used as an evaluation benchmark, while FinCorpus QA from the same Duxiaoman source family is used for CPT and SFT. The paper does not report any overlap analysis between FinCorpus and FinanceIQ, so the high FinanceIQ scores in Table 1 may reflect training-data exposure rather than reasoning ability. The same concern applies to the Duxiaoman FinQA items in SFT and the FinanceIQ evaluation, and the manuscript provides no evidence to rule out leakage.
- [Section 4.2 and Table 1] All results are reported as single-point avg@10 scores with no error bars, no multiple seeds, and no statistical significance testing. The Fin CPA benchmark contains only 215 items, so the reported 10.56-point gap between FEVO-R32B and FEVO-R32B-0 is not robust even in the absence of leakage; with the direct training overlap, the gap cannot support the paper's central validation claim.
minor comments (5)
- [Abstract] The phrase "have lead to" should be "have led to".
- [Section 2.2.3] The sentence "we leverage DeepSeek-R1 to convert to convert single-/multiple-choice questions" contains a duplicated "to convert".
- [Section 3.3.1] The word "thesholds" should be "thresholds".
- [Section 4.1.2] The benchmark name "FIN-EV A" is written with a space; it should be consistent with the dataset name "Fin-Eva" used in Appendix A.2.
- [Section 4.1.2] The paper does not state whether the comparison models were evaluated by the authors under identical settings or whether their scores were taken from published sources; this should be clarified for reproducibility.
Circularity Check
The paper's headline benchmark validation is not independent: the 215-item Fin CPA evaluation set (Appendix A.2) is the same 215-item Fin CPA RL training set (Appendix A.1), and CFLUE evaluation draws on the same benchmark used for SFT, so the reported SOTA and R32B-vs-R32B-0 gaps partly reduce to memorization.
-
fitted input called prediction
[Appendix A.1 (Table 2, RL Data) and Appendix A.2 (Table 3, Evaluation Datasets); abstract and Section 4.2 claim validation]
"RL Data | Fin CPA | Finance | In-house | 215 (items) | Real-world data from accountant exams ... Evaluation Datasets | Fin CPA | In-house | 215 | Gathered from in-house financial tasks."
The 215-item Fin CPA set appears in the RL training data and in the evaluation benchmark with the same name and cardinality and no statement that they are disjoint. FEVO-R32B is therefore trained on the exact questions used to report its Fin CPA score (73.60), and FEVO-R32B-0 (63.04) is also RL-trained on them; the R32B-versus-R32B-0 gap and the 'validation' of CPT/SFT claimed in Section 4.2 are confounded by additional CPT/SFT exposure to the same test material. The headline Fin CPA result is an input to training, not an independent prediction.
-
fitted input called prediction
[Section 2.2.2 / Appendix A.1 (SFT Data) vs. Appendix A.2 (Evaluation Datasets)]
"CFLUE | Finance | Open-source | 20K (items) | Chinese Financial Language Understanding Evaluation ... CFLUE | Open-source | 3864 | Chinese Financial Language Understanding Evaluation (CFLUE), a comprehensive evaluation for LLM's ability in the Chinese financial domain."
SFT training uses 20K items from the public CFLUE benchmark while evaluation reports a 3,864-item CFLUE score from the same benchmark, and RL uses fill-in-blank items 'Extracted from CFLUE' converted to open-ended questions. No train/eval split, n-gram filtering, or decontamination is reported. Consequently the reported CFLUE score is not shown to measure generalization; it may reflect memorization of SFT or RL training material, weakening the claimed state-of-the-art validation.
full rationale
The paper's central derivation—that FEVO-R32B's state-of-the-art financial scores and its advantage over FEVO-R32B-0 validate financial knowledge expansion and structured reasoning—runs through the evaluation benchmarks in Section 4.2. As reported, that evaluation is not independent of the training data. Appendix A.1 lists the RL training set 'Fin CPA, In-house, 215 (items)' and Appendix A.2 lists the evaluation benchmark 'Fin CPA, In-house, 215' with no disjointness or decontamination statement; the identical name and cardinality mean the model was fitted to the exact questions on which its Fin CPA score is reported. A similar overlap is documented for CFLUE: 20K CFLUE items are used for SFT while the 3,864-item CFLUE evaluation is drawn from the same benchmark, and RL data include CFLUE-derived fill-in-blank conversions. No n-gram, MinHash, embedding, or other contamination check is described. Under these conditions the headline benchmark numbers and the R32B-versus-R32B-0 gap are, at least in part, a measure of memorization rather than of the framework's claimed generalization, so the central validation claim reduces to a trained-on-eval input rather than an independent prediction. No self-citation circularity was found; the circularity is in the train/eval construction.
Assumptions & free parameters
free parameters (5)
- Balanced batching thresholds and limits =
m_hard=0.0, m_easy=0.95, L_hard=0.1x, L_easy=0.3x
- Reward weights =
format penalty -2.0, wrong -1.5, correct +2.0, language penalty -0.5, k=0.8
- Length-adaptive GAE alpha =
0.16
- Value model checkpoint selection =
loss < 1 and explained variance > 0.5
- CPT and SFT learning rates and epochs =
CPT lr 3e-4, SFT lr 5e-5, both 1 epoch
assumptions (6)
- domain assumption DeepSeek-R1 and Qwen2.5-72B-Instruct generated reasoning traces are correct enough for training after filtering.
- domain assumption Rule-based filters and GPT-4o judge correctly identify high-quality reasoning traces.
- domain assumption Open-ended conversion preserves the solvability and information content of the original questions.
- domain assumption Evaluation benchmarks do not overlap with training data.
- domain assumption Reported numbers from other models are comparable to the authors' evaluation setup.
- standard math Standard RL and GAE mathematics applies as referenced.
Cite this review
Pith. "Pith review of FEVO: Financial Knowledge Expansion and Reasoning Evolution for Large Language Models." pith.science (2026). https://pith.science/paper/OM3EEZWG
@misc{pith2026250706057,
author = {Pith},
title = {Pith review of: FEVO: Financial Knowledge Expansion and Reasoning Evolution for Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/OM3EEZWG}},
note = {Machine review of arXiv:2507.06057}
}
read the original abstract
Advancements in reasoning for large language models (LLMs) have lead to significant performance improvements for LLMs in various fields such as mathematics and programming. However, research applying these advances to the financial domain, where considerable domain-specific knowledge is necessary to complete tasks, remains limited. To address this gap, we introduce FEVO (Financial Evolution), a multi-stage enhancement framework developed to enhance LLM performance in the financial domain. FEVO systemically enhances LLM performance by using continued pre-training (CPT) to expand financial domain knowledge, supervised fine-tuning (SFT) to instill structured, elaborate reasoning patterns, and reinforcement learning (RL) to further integrate the expanded financial domain knowledge with the learned structured reasoning. To ensure effective and efficient training, we leverage frontier reasoning models and rule-based filtering to curate FEVO-Train, high-quality datasets specifically designed for the different post-training phases. Using our framework, we train the FEVO series of models - C32B, S32B, R32B - from Qwen2.5-32B and evaluate them on seven benchmarks to assess financial and general capabilities, with results showing that FEVO-R32B achieves state-of-the-art performance on five financial benchmarks against much larger models as well as specialist models. More significantly, FEVO-R32B demonstrates markedly better performance than FEVO-R32B-0 (trained from Qwen2.5-32B-Instruct using only RL), thus validating the effectiveness of financial domain knowledge expansion and structured, logical reasoning distillation
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Analysis of Option (1): Impact of changes in an enterprise's own credit risk for financial liabilities designated as measured at fair value through profit or loss: According to relevant regulations, for financial liabilities designated as measured at fair value through profit or loss, the portion of fair value changes due to changes in the enterprise's own cr...
-
[2]
Analysis of Option (2): The effective portion of gains arising from cash flow hedging instruments: According to accounting standards, for cash flow hedges, the gains or losses from the effective portion of the hedge should be recognized in other comprehensive income (OCI). If the hedged forecast transaction ultimately occurs, these gains or losses will be rec...
-
[3]
URLhttps://arxiv.org/abs/2501.12599. Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei L ¨u, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuan- hai Dong, Yanqi Sun, Yifu Chen, Yongyi Peng, Xiaojuan...
arXiv 2023
-
[4]
Analysis of Option (4): Actuarial losses arising from the remeasurement of net defined benefit liabilities: According to relevant regulations, actuarial gains and losses arising from the remeasurement of net defined benefit liabilities or assets should be recognized in other comprehensive income (OCI) and cannot be reclassified to profit or loss in subsequent p...
work page 2023
-
[7]
Analysis of Option (3): Reclassification of investment property from the cost model to the fair value model, where the fair value on the conversion date exceeds the original carrying amount: According to accounting standards, a change in the subsequent measurement model for investment property from the cost model to the fair value model is considered a cha...
-
[9]
Item (1): A financial liability designated as measured at fair value through profit or loss caused a net asset change of 3 million yuan due to changes in the enterprise's own credit risk. According to accounting standards, I recall that in this situation, the fair value change of a financial liability arising from changes in its own credit risk should be rec...
-
[10]
Item (2): The effective portion of gains from cash flow hedging instruments, amounting to 2.8 million yuan. According to accounting standards, the effective portion of a cash flow hedge should be recognized in other comprehensive income. When the hedged forecast transaction affects profit or loss, then the other comprehensive income will be reclassified to profit...
-
[11]
Item (3): Investment property previously measured using the cost model was reclassified to the fair value model, and on the date of conversion, its fair value exceeded its original carrying amount by 4.4 million yuan. In this case, should the difference arising from the conversion be recognized in retained earnings? Or in other comprehensive income? No, acc...
Show all 12 references
-
[12]
According to the treatment of defined benefit plans, the remeasurement component, such as actuarial gains and losses, should be recognized in other comprehensive income
Item (4): Actuarial losses of 3.2 million yuan arising from the remeasurement of net defined benefit liabilities. According to the treatment of defined benefit plans, the remeasurement component, such as actuarial gains and losses, should be recognized in other comprehensive incom...
-
[2018]
formula. For an output with lengthl, the correspondingλ actor as well as the advantage estimate ˆAt at each time step are calculated as follows: λactor = 1− 1 αl (4) ˆAt = l−t−1X k=0 (γλactor)k (R(st, at) +γV(s t+1 −V(s t))(5) A.4 SAMPLE RESPONSES In 2x23, Company A had the fo...
-
[2024]
Duxiaoman DI Team
URLhttps://arxiv.org/abs/2402.00159. Duxiaoman DI Team. Fincorpus.https://huggingface.co/datasets/ Duxiaoman-DI/FinCorpus/tree/main/data, 2023a. Duxiaoman DI Team. Financeiq.https://huggingface.co/datasets/ Duxiaoman-DI/FinanceIQ, 2023b. Duxiaoman DI Team. Xuanyuan-finx1-previ...
2024 arXiv
-
[2025]
Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna
URLhttps://arxiv.org/abs/2501.12948. Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective, 2021. URL https://arxiv.org/abs/1908.04734. Leo Gao, John Schulman, ...
2021 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.