Pith. sign in

REVIEW 3 major objections 5 minor 20 references

LLMs Meet Finance: Fine-Tuning Foundation Models for the Open FinLLM Leaderboard

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning small open language models on financial benchmarks produces large task gains, and the measured financial data-scaling exponent matches the value derived from general-text and math scaling laws.

desk verdict A useful fine-tuning recipe for small financial LLMs with a genuinely new but fragile data-scaling exponent; the universality claim needs a corrected derivation and error bars before it can carry weight. read the letter →

arxiv 2504.13125 v1 pith:GVM44LGR submitted 2025-04-17 cs.CL cs.AIcs.LG

Varun Rao , Youran Sun , Mahendra Kumar , Tejas Mutneja , Agastya Mukherjee , Haizhao Yang This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords largelanguagemodelsfinancialNLPsupervisedfine-tuningdirectpreferenceoptimizationreinforcementlearningdatascalinglawssyntheticOpenFinLLMLeaderboard
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that small, open 1.5B language models can be made strong at financial tasks through a practical fine-tuning recipe, and that financial training data follow the same scaling law seen in other domains. The authors fine-tune Qwen2.5 and DeepSeek-R1 base models with supervised fine-tuning, direct preference optimization, and a synthetic-data loop built on chain-of-thought annotation. They report large gains on Open FinLLM Leaderboard tasks, for example lifting DeepSeek's NER F1 from 0.1448 to 0.7231. They also measure a financial data-scaling exponent of 0.28 and argue it is consistent with a literature-derived exponent of 0.24, which would imply that the scaling exponent is independent of task domain. If true, this matters because it would make finance LLM development cheaper and more predictable from small-scale experiments.

What carries the argument

Two pieces carry the argument. The first is the fine-tuning pipeline: SFT adapts a base 1.5B model to the target format; DPO then uses the SFT model's overly long repetitive outputs as rejected examples, so the model learns to stop at the correct answer; and for datasets without training labels, the model generates its own chain-of-thought annotations, extracts answers with regular expressions, and retrains on the synthesized pairs. The second is the scaling-law identity in Eq. (1), which combines $L\sim p^{-0.076}$, $L\sim d^{-0.095}$, and $E\sim p^{-0.195}$ into $E\sim d^{0.24}$ by taking test error to be a fixed power of loss; the measured exponent 0.28 is compared with that derived value to support the universality claim.

What would settle it

Refit the data-scaling curve separately per task with many more data fractions, say 12.5%, 6%, 3%, and 1.5%, with repeated runs to obtain error bars; if per-task exponents spread widely or the pooled exponent moves significantly away from 0.24, the universality claim is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that 1.5B open LLMs can be turned into competitive financial-task models with a three-stage recipe: supervised fine-tuning, direct preference optimization, and a synthetic-data loop driven by chain-of-thought annotation. SFT on the NER task raises DeepSeek-R1-1.5B's F1 from 0.1448 to 0.7231, and on FiQASA from 0.4383 to 0.7865. Applying DPO afterward cuts the overlength ratio from 54.7% to 1.7% while roughly preserving the trained-task F1 and improving the unseen CC task. For tasks with no official training split, one iteration of synthetic-data training gives a +87.1% F1 gain on MultiFin, +22.5% on FOMC, and a +3.4% RMSE improvement on TSA. The paper further claims a financial data-scaling exponent of 0.28, close to the 0.24 derived by combining prior loss and error scaling laws, and reads this agreement as evidence that the exponent is task-independent.

Load-bearing premise

The load-bearing premise is the bridge in Eq. (1): test error is a fixed power of training loss and parameter count can be exchanged for data size along a constant-loss curve; if that bridge fails, the measured exponent 0.28 cannot be compared with the derived 0.24 as evidence of universality.

Editorial extensions

If this is right

  • Fine-tuning with LoRA on 1.5B models can reach practical score levels on specialized finance benchmarks, so strong finance LLM performance does not require large proprietary models.
  • DPO after SFT repairs repetitive generation and can improve an unseen classification task, making preference tuning a useful general repair step in small-model fine-tuning.
  • A synthetic-data loop lets teams train on leaderboard tasks that publish no training split, and even one iteration yields large gains on some tasks.
  • If the financial data-scaling exponent is near 0.24, practitioners can extrapolate the benefit of additional training data from a few small runs, which would make leaderboard development much cheaper.
  • Cross-task transfer is not uniform: fine-tuning on NER can hurt causal classification, while fine-tuning on FiQASA helps financial polarity sentiment, so the choice of fine-tuning data matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A per-task scaling-law fit with more data fractions and repeated runs would test whether the 0.28 exponent is a genuine universal constant or an average over heterogeneous task exponents; the paper's aggregate curve cannot distinguish these.
  • The synthetic-data loop is self-training, so part of its measured gain could be the model learning its own annotation biases; comparing synthetic-only training with gold-label training on tasks where gold labels exist would quantify that contamination.
  • Because DPO targets overly long outputs, similar fixes may be achievable at inference time through repetition penalties or sampling changes; the paper mentions this possibility in its discussion but does not test it.
  • If the 0.28-versus-0.24 agreement survives more careful measurement, it connects finance NLP to a physics-style critical-phenomenon picture of LLM training; that picture is an interpretation the paper offers, not something its experiments establish on their own.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports supervised fine-tuning (SFT), direct preference optimization (DPO), and a pipeline here labeled reinforcement learning (RL) with model-synthesized data to adapt 1.5B-parameter open LLMs (Qwen2.5-1.5B-Instruct and DeepSeek-R1-1.5B) to financial tasks from the Open FinLLM Leaderboard. The authors report large improvements on several tasks, e.g., DeepSeek NER F1 from 0.1448 to 0.7231 after SFT, and a +87.1% relative gain on MultiFin with synthetic data. The final experimental section fits a data-scaling curve with exponent 0.28 and claims this is highly consistent with a literature-derived value of 0.24, arguing for universality of the scaling exponent across domains. The central claims are therefore empirical gains from fine-tuning and the existence of a financial-domain data scaling law; the first claim is supported by the tables, while the second rests on a fragile derivation and a four-point fit.

Significance. If the fine-tuning results are taken at face value, they provide useful evidence that instruction-tuned 1.5B models can be substantially improved on financial benchmarks with modest compute, and the comparison of SFT, DPO, and synthetic-data variants is a practical contribution. The paper also reports hyperparameters and training configurations, which aids reproducibility. The scaling-law claim would be significant if established, since cross-domain universality of the data-scaling exponent is an interesting hypothesis; however, as presented in Section IV-d, the derivation in Eq. (1) and the four-point aggregate fit with no error bars or seeds do not currently support the claimed agreement with 0.24. The paper's strength is the internally consistent table of fine-tuning results; its weakness is the load-bearing scaling-law comparison.

major comments (3)
  1. [Section IV-d, Eq. (1)] The derivation of E ~ d^{0.24} is not established. Eq. (1) combines L ~ p^{-0.076} and L ~ d^{-0.095} from [14] with E ~ p^{-0.195} from [15], which requires (a) that the variable p in [15] is model parameter count rather than inference-time compute, and (b) that test error E is a fixed power of loss L with exponent 2.57 throughout the relevant regime. Neither condition is shown, and [15] is an inference-scaling study, so the variable mismatch is plausible. The algebra also appears to drop a minus sign: if E ~ L^{2.57} and L ~ d^{-0.095}, then E ~ d^{-0.24}, not d^{0.24}; the sign convention in the comparison to the fitted exponent 0.28 should be clarified and the derivation made explicit.
  2. [Section IV-d, Fig. 2] The empirical scaling exponent is fitted to only four data fractions (1, 1/2, 1/4, 1/8) with no seeds, no confidence intervals, and no statement about how data fractions were stratified across the heterogeneous financial tasks. One outlier point can materially change the fitted exponent, so the statement that 0.28 is 'very close' to 0.24 overstates the support. The authors should report per-task scaling fits, standard deviations across seeds or bootstrap intervals, and the number of tasks contributing at each fraction, or soften the universality claim accordingly.
  3. [Sections III-c and IV-c; Table III] The procedure labeled 'reinforcement learning (RL)' is actually SFT and DPO on model-synthesized data: the five steps described are corpus collection, LLM annotation via chain-of-thought, answer extraction, query formatting, and SFT/DPO training. There is no RL loss, reward model, trajectory sampling, or policy-gradient update. Calling this pipeline RL is inaccurate and affects the abstract, the introduction, and contribution 2. The authors should either rename it 'synthetic-data fine-tuning' throughout or provide an actual RL formulation. Additionally, Table III reports only relative performance boosts; absolute scores and the base-model performance for each task should be reported.
minor comments (5)
  1. [Introduction and Section III-a] The number of datasets is inconsistent: the introduction says the leaderboard contains 36 datasets over 24 tasks, while Section III-a says 'among all 41 datasets, 28 provide training/validation data.' Please reconcile the counts.
  2. [Figure 2] The axis labels and fitted equation are malformed: the text shows '11/21/41/8' and 'log(1 F1) = 0.28 × log(d) 1.35', which is missing a minus sign before 1.35 and is hard to read. The variable d (data fraction) and the y-axis quantity should be printed clearly, and the sign of the exponent should be consistent with the direction of the scaling relation.
  3. [Section IV-c, heading] The heading 'RL with Synthesed Data' contains a typo: 'synthesed' should be 'synthesized'.
  4. [Section IV-b, Table II] Table II shows that DPO reduces the overlength ratio and improves the unseen CC task, but the NER F1 decreases slightly; the text says this 'remains unchanged mainly' and attributes the decrease to the scoring mechanism. Please provide a more concrete explanation or report whether the difference is within evaluation noise.
  5. [Section IV-d] The authors use the term 'data critical exponent' for the slope of log(1-F1) versus log(data fraction). This terminology is unusual; consider using 'data-scaling exponent' and defining the exact quantity that is being fitted.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation found: the central gains and exponent comparison are external-benchmark or external-literature based; the only self-citation is a non-load-bearing universality reference.

full rationale

The paper's main empirical claims are direct evaluations on the held-out Open FinLLM Leaderboard tasks after supervised fine-tuning, DPO, or synthetic-data training, so the reported gains are not equivalent to the training inputs by construction. The data-scaling exponent 0.28 is fitted to the paper's own four-fraction aggregate average-F1 curve, and the comparison value 0.24 is derived from external references [14] and [15]; although that comparison is statistically fragile because it lacks error bars, seeds, and per-task stratification, it is not circular because the 0.24 target does not come from the paper's own fit. The only self-citation is reference [16], by co-author Y. Sun, used in the sentence 'it exhibits a form of universality [16].' That citation supplies no derivational step and the universality claim rests on the external exponent agreement, so it is a minor self-citation that is not load-bearing. No step in the paper reduces a prediction to its inputs by definition, by fitted-parameter renaming, or by a load-bearing self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central scaling claim rests on a fitted power law, on a cross-domain conversion of loss exponents to error exponents that is not proven, and on treating leaderboard scores as valid measures of financial ability. The fine-tuning claims rest on standard SFT and DPO assumptions and on self-generated synthetic data.

free parameters (4)
  • scaling exponent alpha = 0.28
    Fitted to log(1-F1) versus log(data fraction) across four points in Section IV-d and Figure 2; this is the number compared with 0.24.
  • scaling intercept C = -1.35
    Fitted intercept of the same line; needed to specify the scaling curve.
  • LoRA rank and alpha = rank 128, alpha 256 for SFT; rank 16, alpha 32 for DPO
    Hand-chosen hyperparameters affecting all fine-tuning results; not used in the scaling-law derivation.
  • DPO temperature beta = 1
    Standard temperature hyperparameter in the DPO loss, set to 1 and not tuned.
assumptions (4)
  • domain assumption Average error follows a power law in data fraction
    The paper assumes log(1-F1) = alpha log(d) + C without deriving this from a model; it is an empirical modeling assumption (Section IV-d, Figure 2).
  • ad hoc to paper Error rate E is a power of loss L, giving E ~ L^2.57
    Eq. (1) combines Kaplan's loss exponents with Wu's error-rate exponent, implicitly assuming E = c L^gamma with gamma = 2.57 and a constant-loss p-d tradeoff; neither cited paper establishes this relation.
  • domain assumption Open FinLLM Leaderboard scores are valid proxies for financial capability
    The paper treats F1 and RMSE on benchmark tasks as the target measure; no evidence links these to real financial decision quality.
  • domain assumption Synthetic annotations generated by the model itself are reliable training targets
    The RL pipeline uses DeepSeek-generated CoT answers with regex extraction as ground truth; no human validation or quality filter is described (Section III-c).

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs Meet Finance: Fine-Tuning Foundation Models for the Open FinLLM Leaderboard." pith.science (2026). https://pith.science/paper/GVM44LGR

@misc{pith2026250413125,
  author       = {Pith},
  title        = {Pith review of: LLMs Meet Finance: Fine-Tuning Foundation Models for the Open FinLLM Leaderboard},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GVM44LGR}},
  note         = {Machine review of arXiv:2504.13125}
}
read the original abstract

This paper investigates the application of large language models (LLMs) to financial tasks. We fine-tuned foundation models using the Open FinLLM Leaderboard as a benchmark. Building on Qwen2.5 and Deepseek-R1, we employed techniques including supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning (RL) to enhance their financial capabilities. The fine-tuned models demonstrated substantial performance gains across a wide range of financial tasks. Moreover, we measured the data scaling law in the financial domain. Our work demonstrates the potential of large language models (LLMs) in financial applications.

Figures

Figures reproduced from arXiv: 2504.13125 by the authors.

Figure 1
Figure 1. Training flowchart showing the progression from base model to final [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The data scaling law on financial tasks. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 5 canonical work pages

  1. [16]

    Phase transitions in large language models and the o(n) model,

    Y . Sun and B. Haghighat, “Phase transitions in large language models and the o(n) model,” 2025. [Online]. Available: https: //arxiv.org/abs/2501.16241

    arXiv 2025

  2. [14]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” 2020. [Online]. Available: https://arxiv.org/abs/2001.08361

    arXiv 2020

  3. [15]

    Available: https://arxiv.org/abs/2403.13372

    [Online]. Available: https://arxiv.org/abs/2403.13372

    arXiv

  4. [1]

    Finben: A holistic financial benchmark for large language models,

    Q. Xie, W. Han, Z. Chen, R. Xiang, X. Zhang, Y . He, M. Xiao, D. Li, Y . Dai, D. Feng, Y . Xu, H. Kang, Z. Kuang, C. Yuan, K. Yang, Z. Luo, T. Zhang, Z. Liu, G. Xiong, Z. Deng, Y . Jiang, Z. Yao, H. Li, Y . Yu, G. Hu, J. Huang, X.-Y . Liu, A. Lopez-Lira, B. Wang, Y . Lai, H. Wang, M. Peng, S. Ananiadou, and J. Huang, “Finben: A holistic financial benchmar...

    arXiv 2024

  5. [2]

    Fnspid: A comprehensive financial news dataset in time series,

    Z. Dong, X. Fan, and Z. Peng, “Fnspid: A comprehensive financial news dataset in time series,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 4918–4927. [Online]. Available: https://doi.org/10.1145/3637528.3671629

    arXiv 2024

  6. [3]

    Finrl-meta: Market environments and benchmarks for data-driven financial reinforcement learning,

    X.-Y . Liu, Z. Xia, J. Rui, J. Gao, H. Yang, M. Zhu, C. D. Wang, Z. Wang, and J. Guo, “Finrl-meta: Market environments and benchmarks for data-driven financial reinforcement learning,” 2022. [Online]. Available: https://arxiv.org/abs/2211.03107

    arXiv 2022

  7. [4]

    Dynamic datasets and market environments for financial reinforcement learning,

    X.-Y . Liu, Z. Xia, H. Yang, J. Gao, D. Zha, M. Zhu, C. D. Wang, Z. Wang, and J. Guo, “Dynamic datasets and market environments for financial reinforcement learning,” 2023. [Online]. Available: https://arxiv.org/abs/2304.13174

    arXiv 2023

  8. [5]

    Finrl-deepseek: Llm-infused risk-sensitive reinforcement learning for trading agents,

    M. Benhenda, “Finrl-deepseek: Llm-infused risk-sensitive reinforcement learning for trading agents,” 2025. [Online]. Available: https: //arxiv.org/abs/2502.07393

    arXiv 2025

Show all 20 references
  1. [6]

    A report on financial regulations challenge at coling 2025,

    K. Wang, J. Patel, C. Shen, D. Kim, A. Zhu, A. Lin, L. Borella, C. Osborne, M. White, S. Yang, K. Xiao, and X.-Y . L. Yanglet, “A report on financial regulations challenge at coling 2025,” 2025. [Online]. Available: https://arxiv.org/abs/2412.11159

    2025 arXiv

  2. [7]

    FinMind-Y-me at the regulations challenge task: Financial mind your meaning based on THaLLE,

    P. Chantangphol, P. Balee, K. Sucharitpongpan, C. Saetia, and T. Chalothorn, “FinMind-Y-me at the regulations challenge task: Financial mind your meaning based on THaLLE,” in Proceedings of the Joint Workshop of the 9th Financial Technology and Natural Language Processing (Fin...

    2025

  3. [8]

    Simulating financial market via large language model based agents,

    S. Gao, Y . Wen, M. Zhu, J. Wei, Y . Cheng, Q. Zhang, and S. Shang, “Simulating financial market via large language model based agents,”

  4. [9]

    EconAgent: Large language model-empowered agents for simulating macroeconomic activities,

    N. Li, C. Gao, M. Li, Y . Li, and Q. Liao, “EconAgent: Large language model-empowered agents for simulating macroeconomic activities,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) , L.-W. Ku, A. Martins, an...

    2024

  5. [10]

    The curious case of neural text degeneration,

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi, “The curious case of neural text degeneration,” 2020. [Online]. Available: https://arxiv.org/abs/1904.09751

    2020 arXiv

  6. [11]

    Learning dynamics of llm finetuning,

    Y . Ren and D. J. Sutherland, “Learning dynamics of llm finetuning,”

  7. [12]

    Llamafactory: Unified efficient fine-tuning of 100+ language models,

    Y . Zheng, R. Zhang, J. Zhang, Y . Ye, Z. Luo, Z. Feng, and Y . Ma, “Llamafactory: Unified efficient fine-tuning of 100+ language models,”

  8. [13]

    Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models,

    S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang, S. L. Song, S. Rajbhandari, and Y . He, “Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models,”

  9. [19]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models,

    Y . Wu, Z. Sun, S. Li, S. Welleck, and Y . Yang, “Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models,” 2024. [Online]. Available: https://arxiv.org/abs/2408.00724

    2024 arXiv

  10. [2023]

    Available: https://arxiv.org/abs/2309.14509

    [Online]. Available: https://arxiv.org/abs/2309.14509

    arXiv

  11. [2024]

    Available: https://arxiv.org/abs/2406.19966

    [Online]. Available: https://arxiv.org/abs/2406.19966

    arXiv

  12. [2025]

    Available: https://arxiv.org/abs/2407.10490

    [Online]. Available: https://arxiv.org/abs/2407.10490

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.