Pith. sign in

REVIEW 4 major objections 5 minor 12 references

FEVO: Financial Knowledge Expansion and Reasoning Evolution for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read FEVO-R32B, a 32-billion-parameter model trained with continued pre-training, supervised fine-tuning, and reinforcement learning, is claimed to achieve state-of-the-art accuracy on five Chinese financial benchmarks, beating much larger…

desk verdict A transparent, well-built financial LLM pipeline whose SOTA claim is not interpretable: the 215-item Fin CPA evaluation set matches the RL training set in name and size, so the headline numbers are likely memorization until held-out evaluation and decontamination are provided. read the letter →

arxiv 2507.06057 v2 pith:OM3EEZWG submitted 2025-07-08 cs.AI cs.LG

classification cs.AIcs.LG
keywords financialreasoningcontinuedpre-trainingstructuredchain-of-thoughtreinforcementlearningrewardhackingbenchmarksChineseexamsknowledgedistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes FEVO, a three-stage post-training pipeline—continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL)—to make a 32-billion-parameter open-weight language model expert-level at Chinese financial and taxation reasoning. The central claim is that the final model, FEVO-R32B, achieves state-of-the-art results on five financial benchmarks, beating much larger general-purpose models and finance-specialist models, and that the full pipeline clearly outperforms a control model trained with only RL. The value, if true, is that a medium-scale open model can approach domain-expert performance in finance without needing a larger model, and that the three-stage recipe plus careful data curation is the cause. The paper also contributes a data-processing pipeline that converts multiple-choice questions into open-ended ones, which it argues reduces reward hacking in RL.

What carries the argument

The load-bearing mechanism is the sequential three-stage training recipe. First, CPT expands the model's financial knowledge by training on about 188M tokens of financial textbooks, exam questions, and general data. Second, SFT instills a structured reasoning pattern—plan, reasoning, reflection, backtracking, answer—by fine-tuning on 291K chain-of-thought traces distilled from a frontier reasoning model. Third, RL integrates the two using a modified version of the VAPO (Value-Augmented Proximal Policy Optimization) algorithm with rule-based rewards, balanced batching, and a language-consistency penalty. The designed object that carries the argument is the FEVO-Train dataset suite: each stage has its own filtered dataset, and the RL set is made of open-ended questions (converted from multiple-choice questions) to prevent guessing and reward hacking.

What would settle it

Evaluate FEVO-R32B and FEVO-R32B-0 on a newly assembled set of Chinese financial exam questions that were written after the model's training data cutoff and never used in any stage of the FEVO pipeline, including the MCQ-to-open-ended conversion step. If the accuracy gap between R32B and R32B-0 collapses, or if R32B's scores fall below the state-of-the-art levels reported on the original benchmarks, the central claim that the pipeline teaches transferable financial reasoning would be falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that expanding a 32B model's financial knowledge before teaching it structured reasoning, and then integrating the two by reinforcement learning, yields a model that outperforms much larger and finance-specialized competitors on Chinese financial and taxation benchmarks. Specifically, FEVO-R32B is reported to reach state-of-the-art accuracy on Fin CPA, Fin CCR, FinanceIQ, CFLUE, and FIN-EVA, and to markedly outperform FEVO-R32B-0, the same base model trained with only RL. The paper interprets the gap between the two models as direct evidence that the CPT and SFT stages contribute real capability beyond what RL alone induces.

Load-bearing premise

The reported superiority of FEVO-R32B over its RL-only control and over other models assumes that the evaluation benchmarks were not used during training; the in-house 215-item Fin CPA set appears in both the RL training data (Appendix A.1) and the evaluation set (Appendix A.2), and CFLUE and FinanceIQ derive from the same open sources used for training.

Editorial extensions

If this is right

  • If the reported results hold, a 32B open-weight model can match or exceed much larger closed models and finance-specialist systems on Chinese financial exams, lowering the cost of deploying domain-expert-level financial reasoning.
  • The comparison against the RL-only control (FEVO-R32B-0) implies that continued pre-training and structured SFT contribute substantial capability gains on top of RL, not just training noise.
  • Converting multiple-choice questions to open-ended ones is presented as a way to curb reward hacking and expand the size of usable RL training data.
  • Balanced batching in the RL stage is reported to speed up training by about 20% without degrading performance, making the pipeline more practical.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported SOTA numbers on Fin CPA, CFLUE, and FinanceIQ may partly reflect benchmark leakage: the 215-item Fin CPA set is used in both RL training and evaluation, and CFLUE/FinanceIQ come from the same open sources used for training; a truly held-out evaluation set would be needed to confirm transfer.
  • A direct test of the framework's generality would be to apply the same CPT-SFT-RL recipe to another knowledge-heavy domain (e.g., law or medicine) and see if the pattern of gains over an RL-only control repeats.
  • The claim that the gap between FEVO-R32B and FEVO-R32B-0 comes from knowledge expansion and structured reasoning could be further probed by ablating one stage at a time (CPT only + RL, or SFT only + RL) on a contamination-free benchmark.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FEVO, a three-stage post-training framework (CPT, SFT, RL) built on Qwen2.5-32B for financial-domain tasks, and curates stage-specific datasets FEVO-Train-C/S/R. The authors report that the resulting FEVO-R32B achieves state-of-the-art accuracy on five financial benchmarks compared to larger general models and financial specialists, and that its margin over FEVO-R32B-0 validates the contribution of the CPT and SFT stages. The evaluation evidence, however, is compromised by direct overlap between training and evaluation data: the 215-item in-house Fin CPA set appears both in RL training (Appendix A.1) and in the evaluation benchmark (Appendix A.2), and the CFLUE and FinanceIQ benchmarks overlap in source or content with training corpora. The paper reports no decontamination checks, so the headline results and the R32B-versus-R32B-0 gap may be memorization artifacts rather than measures of generalization.

Significance. If the reported results were valid, FEVO would be a meaningful contribution: the pipeline is clearly described, the ablations across C32B/S32B/R32B and R32B-0 are well designed, and the engineering details (balanced batching, reward shaping, open-ended conversion for RL) are useful for practitioners. The central claim, however, is that benchmark scores measure generalization, and that condition is not met in the current manuscript. The direct train/evaluation overlap on Fin CPA and the plausible overlap through CFLUE and FinanceIQ invalidate the state-of-the-art claim and the validation of the framework's components. Without a decontaminated evaluation on held-out data, the experimental evidence cannot support the abstract's conclusions.

major comments (4)
  1. [Appendix A.1/A.2, Tables 2 and 3] The in-house Fin CPA set is listed as RL training data with 215 items in Table 2 and as an evaluation benchmark with 215 items in Table 3, with no statement that the two sets are disjoint. This means the model is trained on the exact questions on which it is evaluated. Consequently, the Fin CPA column in Table 1 (FEVO-R32B 73.60 vs. FEVO-R32B-0 63.04) cannot be interpreted as evidence of generalization, and the claim of state-of-the-art performance on Fin CPA is unsupported.
  2. [Section 2.2.3 and Appendix A.1/A.2] The RL data include "Dianjin Fill-in-blanks" described as "Extracted from CFLUE (no overlap with SFT data)", while the SFT data include 20K CFLUE items and the evaluation uses the CFLUE benchmark with 3,864 items. The parenthetical only asserts disjointness between two training stages; it does not rule out overlap with the evaluation set. Since the paper reports no n-gram, MinHash, or embedding decontamination against the evaluation benchmarks, the CFLUE results in Table 1 are not a valid measure of out-of-training generalization.
  3. [Section 2.1 and Appendix A.2] FinanceIQ (Duxiaoman) is used as an evaluation benchmark, while FinCorpus QA from the same Duxiaoman source family is used for CPT and SFT. The paper does not report any overlap analysis between FinCorpus and FinanceIQ, so the high FinanceIQ scores in Table 1 may reflect training-data exposure rather than reasoning ability. The same concern applies to the Duxiaoman FinQA items in SFT and the FinanceIQ evaluation, and the manuscript provides no evidence to rule out leakage.
  4. [Section 4.2 and Table 1] All results are reported as single-point avg@10 scores with no error bars, no multiple seeds, and no statistical significance testing. The Fin CPA benchmark contains only 215 items, so the reported 10.56-point gap between FEVO-R32B and FEVO-R32B-0 is not robust even in the absence of leakage; with the direct training overlap, the gap cannot support the paper's central validation claim.
minor comments (5)
  1. [Abstract] The phrase "have lead to" should be "have led to".
  2. [Section 2.2.3] The sentence "we leverage DeepSeek-R1 to convert to convert single-/multiple-choice questions" contains a duplicated "to convert".
  3. [Section 3.3.1] The word "thesholds" should be "thresholds".
  4. [Section 4.1.2] The benchmark name "FIN-EV A" is written with a space; it should be consistent with the dataset name "Fin-Eva" used in Appendix A.2.
  5. [Section 4.1.2] The paper does not state whether the comparison models were evaluated by the authors under identical settings or whether their scores were taken from published sources; this should be clarified for reproducibility.

Circularity Check

2 steps flagged · score 8.0 of 10

The paper's headline benchmark validation is not independent: the 215-item Fin CPA evaluation set (Appendix A.2) is the same 215-item Fin CPA RL training set (Appendix A.1), and CFLUE evaluation draws on the same benchmark used for SFT, so the reported SOTA and R32B-vs-R32B-0 gaps partly reduce to memorization.

  1. fitted input called prediction [Appendix A.1 (Table 2, RL Data) and Appendix A.2 (Table 3, Evaluation Datasets); abstract and Section 4.2 claim validation]
    "RL Data | Fin CPA | Finance | In-house | 215 (items) | Real-world data from accountant exams ... Evaluation Datasets | Fin CPA | In-house | 215 | Gathered from in-house financial tasks."

    The 215-item Fin CPA set appears in the RL training data and in the evaluation benchmark with the same name and cardinality and no statement that they are disjoint. FEVO-R32B is therefore trained on the exact questions used to report its Fin CPA score (73.60), and FEVO-R32B-0 (63.04) is also RL-trained on them; the R32B-versus-R32B-0 gap and the 'validation' of CPT/SFT claimed in Section 4.2 are confounded by additional CPT/SFT exposure to the same test material. The headline Fin CPA result is an input to training, not an independent prediction.

  2. fitted input called prediction [Section 2.2.2 / Appendix A.1 (SFT Data) vs. Appendix A.2 (Evaluation Datasets)]
    "CFLUE | Finance | Open-source | 20K (items) | Chinese Financial Language Understanding Evaluation ... CFLUE | Open-source | 3864 | Chinese Financial Language Understanding Evaluation (CFLUE), a comprehensive evaluation for LLM's ability in the Chinese financial domain."

    SFT training uses 20K items from the public CFLUE benchmark while evaluation reports a 3,864-item CFLUE score from the same benchmark, and RL uses fill-in-blank items 'Extracted from CFLUE' converted to open-ended questions. No train/eval split, n-gram filtering, or decontamination is reported. Consequently the reported CFLUE score is not shown to measure generalization; it may reflect memorization of SFT or RL training material, weakening the claimed state-of-the-art validation.

full rationale

The paper's central derivation—that FEVO-R32B's state-of-the-art financial scores and its advantage over FEVO-R32B-0 validate financial knowledge expansion and structured reasoning—runs through the evaluation benchmarks in Section 4.2. As reported, that evaluation is not independent of the training data. Appendix A.1 lists the RL training set 'Fin CPA, In-house, 215 (items)' and Appendix A.2 lists the evaluation benchmark 'Fin CPA, In-house, 215' with no disjointness or decontamination statement; the identical name and cardinality mean the model was fitted to the exact questions on which its Fin CPA score is reported. A similar overlap is documented for CFLUE: 20K CFLUE items are used for SFT while the 3,864-item CFLUE evaluation is drawn from the same benchmark, and RL data include CFLUE-derived fill-in-blank conversions. No n-gram, MinHash, embedding, or other contamination check is described. Under these conditions the headline benchmark numbers and the R32B-versus-R32B-0 gap are, at least in part, a measure of memorization rather than of the framework's claimed generalization, so the central validation claim reduces to a trained-on-eval input rather than an independent prediction. No self-citation circularity was found; the circularity is in the train/eval construction.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several unverified assumptions about data quality and contamination. The most serious is the assumption that evaluation benchmarks are disjoint from training data, which the paper's own appendices contradict. The free parameters are mostly RL hyperparameters that are chosen by hand and not systematically varied. There are no invented physical or conceptual entities.

free parameters (5)
  • Balanced batching thresholds and limits = m_hard=0.0, m_easy=0.95, L_hard=0.1x, L_easy=0.3x
    Hand-chosen to stabilize RL training; they directly shape the training batch and final model but are not ablated.
  • Reward weights = format penalty -2.0, wrong -1.5, correct +2.0, language penalty -0.5, k=0.8
    Arbitrary scaled rewards, central to the RL optimization, with no sensitivity analysis.
  • Length-adaptive GAE alpha = 0.16
    Hyperparameter for advantage calculation, presumably tuned during preliminary experiments; no ablation reported.
  • Value model checkpoint selection = loss < 1 and explained variance > 0.5
    Checkpoint chosen from preliminary RL runs; this affects policy training quality but is not justified in detail.
  • CPT and SFT learning rates and epochs = CPT lr 3e-4, SFT lr 5e-5, both 1 epoch
    Standard settings, but no ablation shows they are optimal for this pipeline.
assumptions (6)
  • domain assumption DeepSeek-R1 and Qwen2.5-72B-Instruct generated reasoning traces are correct enough for training after filtering.
    Data quality depends on frontier model outputs; errors introduced during generation could propagate into the fine-tuned model.
  • domain assumption Rule-based filters and GPT-4o judge correctly identify high-quality reasoning traces.
    There is no validation of judge accuracy or filter precision and recall in the paper.
  • domain assumption Open-ended conversion preserves the solvability and information content of the original questions.
    The conversion is model-generated and only checked with the listed criteria; no formal or human verification is reported.
  • domain assumption Evaluation benchmarks do not overlap with training data.
    This assumption is contradicted by Fin CPA appearing in both training and evaluation, and CFLUE/FinanceIQ sources also being used in training.
  • domain assumption Reported numbers from other models are comparable to the authors' evaluation setup.
    Different tools, prompts, temperatures, and model versions across papers can change scores; no standardized comparison is shown.
  • standard math Standard RL and GAE mathematics applies as referenced.
    The paper uses VAPO and PPO formulas from cited works, which are treated as background.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FEVO: Financial Knowledge Expansion and Reasoning Evolution for Large Language Models." pith.science (2026). https://pith.science/paper/OM3EEZWG

@misc{pith2026250706057,
  author       = {Pith},
  title        = {Pith review of: FEVO: Financial Knowledge Expansion and Reasoning Evolution for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OM3EEZWG}},
  note         = {Machine review of arXiv:2507.06057}
}
read the original abstract

Advancements in reasoning for large language models (LLMs) have lead to significant performance improvements for LLMs in various fields such as mathematics and programming. However, research applying these advances to the financial domain, where considerable domain-specific knowledge is necessary to complete tasks, remains limited. To address this gap, we introduce FEVO (Financial Evolution), a multi-stage enhancement framework developed to enhance LLM performance in the financial domain. FEVO systemically enhances LLM performance by using continued pre-training (CPT) to expand financial domain knowledge, supervised fine-tuning (SFT) to instill structured, elaborate reasoning patterns, and reinforcement learning (RL) to further integrate the expanded financial domain knowledge with the learned structured reasoning. To ensure effective and efficient training, we leverage frontier reasoning models and rule-based filtering to curate FEVO-Train, high-quality datasets specifically designed for the different post-training phases. Using our framework, we train the FEVO series of models - C32B, S32B, R32B - from Qwen2.5-32B and evaluate them on seven benchmarks to assess financial and general capabilities, with results showing that FEVO-R32B achieves state-of-the-art performance on five financial benchmarks against much larger models as well as specialist models. More significantly, FEVO-R32B demonstrates markedly better performance than FEVO-R32B-0 (trained from Qwen2.5-32B-Instruct using only RL), thus validating the effectiveness of financial domain knowledge expansion and structured, logical reasoning distillation

Figures

Figures reproduced from arXiv: 2507.06057 by the authors.

Figure 1
Figure 1. Overall pipeline for FEVO, including dataset curation and training To address the above challenges, we designed FEVO (Financial Evolution), a framework to enhance LLM performance in the financial domain (as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Curation process for the CPT dataset In the CPT stage, our objective is to expand the trained model’s knowledge (Ke et al., 2023) in the financial domain, thus ensuring that it can recall sufficient amounts of relevent knowledge when solving financial tasks. Thus we utilize open-source financial text corpus to obtain copius amounts of domain data. One major weakness of pretrained models is insufficient keyword expan… view at source ↗
Figure 3
Figure 3. Example of a structured CoT response and the filtering process to obtain SFT dataset [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Example of converting a single-choice question to an open-ended one. Necessary in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: RL training process using VAPO algorithm [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Comparison of a sample response for R32B and R32B-0 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

  1. [1]

    When the financial liability is derecognized, this portion should be reclassified from OCI to profit or loss

    Analysis of Option (1): Impact of changes in an enterprise's own credit risk for financial liabilities designated as measured at fair value through profit or loss: According to relevant regulations, for financial liabilities designated as measured at fair value through profit or loss, the portion of fair value changes due to changes in the enterprise's own cr...

  2. [2]

    If the hedged forecast transaction ultimately occurs, these gains or losses will be reclassified to profit or loss in the future

    Analysis of Option (2): The effective portion of gains arising from cash flow hedging instruments: According to accounting standards, for cash flow hedges, the gains or losses from the effective portion of the hedge should be recognized in other comprehensive income (OCI). If the hedged forecast transaction ultimately occurs, these gains or losses will be rec...

  3. [3]

    URLhttps://arxiv.org/abs/2501.12599. Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei L ¨u, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuan- hai Dong, Yanqi Sun, Yifu Chen, Yongyi Peng, Xiaojuan...

  4. [4]

    Analysis of Option (4): Actuarial losses arising from the remeasurement of net defined benefit liabilities: According to relevant regulations, actuarial gains and losses arising from the remeasurement of net defined benefit liabilities or assets should be recognized in other comprehensive income (OCI) and cannot be reclassified to profit or loss in subsequent p...

  5. [7]

    Analysis of Option (3): Reclassification of investment property from the cost model to the fair value model, where the fair value on the conversion date exceeds the original carrying amount: According to accounting standards, a change in the subsequent measurement model for investment property from the cost model to the fair value model is considered a cha...

  6. [9]

    Item (1): A financial liability designated as measured at fair value through profit or loss caused a net asset change of 3 million yuan due to changes in the enterprise's own credit risk. According to accounting standards, I recall that in this situation, the fair value change of a financial liability arising from changes in its own credit risk should be rec...

  7. [10]

    According to accounting standards, the effective portion of a cash flow hedge should be recognized in other comprehensive income

    Item (2): The effective portion of gains from cash flow hedging instruments, amounting to 2.8 million yuan. According to accounting standards, the effective portion of a cash flow hedge should be recognized in other comprehensive income. When the hedged forecast transaction affects profit or loss, then the other comprehensive income will be reclassified to profit...

  8. [11]

    Item (3): Investment property previously measured using the cost model was reclassified to the fair value model, and on the date of conversion, its fair value exceeded its original carrying amount by 4.4 million yuan. In this case, should the difference arising from the conversion be recognized in retained earnings? Or in other comprehensive income? No, acc...

Show all 12 references
  1. [12]

    According to the treatment of defined benefit plans, the remeasurement component, such as actuarial gains and losses, should be recognized in other comprehensive income

    Item (4): Actuarial losses of 3.2 million yuan arising from the remeasurement of net defined benefit liabilities. According to the treatment of defined benefit plans, the remeasurement component, such as actuarial gains and losses, should be recognized in other comprehensive incom...

  2. [2018]

    formula. For an output with lengthl, the correspondingλ actor as well as the advantage estimate ˆAt at each time step are calculated as follows: λactor = 1− 1 αl (4) ˆAt = l−t−1X k=0 (γλactor)k (R(st, at) +γV(s t+1 −V(s t))(5) A.4 SAMPLE RESPONSES In 2x23, Company A had the fo...

  3. [2024]

    Duxiaoman DI Team

    URLhttps://arxiv.org/abs/2402.00159. Duxiaoman DI Team. Fincorpus.https://huggingface.co/datasets/ Duxiaoman-DI/FinCorpus/tree/main/data, 2023a. Duxiaoman DI Team. Financeiq.https://huggingface.co/datasets/ Duxiaoman-DI/FinanceIQ, 2023b. Duxiaoman DI Team. Xuanyuan-finx1-previ...

  4. [2025]

    Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna

    URLhttps://arxiv.org/abs/2501.12948. Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective, 2021. URL https://arxiv.org/abs/1908.04734. Leo Gao, John Schulman, ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.