{"id":"7cd938de-4468-45e7-8469-98d7ad876a84","arxiv_id":"2504.13125","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning 1.5B LLMs with SFT, DPO, and synthesized data improves scores on several Open FinLLM Leaderboard tasks, and a fitted data-scaling exponent of 0.28 is claimed to match a cross-domain value of 0.24.","lead":"This paper fine-tunes two small open-source language models (Qwen2.5 and DeepSeek-R1, both 1.5B) on financial benchmarks and reports that SFT, DPO, and synthetic-data training improve task scores. It also fits a data-scaling curve across financial tasks and claims the exponent (0.28) matches a value derived from prior scaling-law literature (0.24).","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scaling-law universality claim rests on Eq. (1), whose cross-paper exponent composition is not justified, and on a four-point fit with no error bars; the 0.28 vs 0.24 agreement is therefore not yet established.","rationale":"Read in good faith, the paper's main empirical contributions (SFT/DPO gains in Tables I-II and synthetic-data gains in Table III) are plausible and clearly presented, though no code or data are released. The unique scientific claim is the data-scaling-law universality in Section IV-d, and the reader's weakest assumption correctly identifies Eq. (1) as the load-bearing step. The derivation combines separate fits from two papers under an unstated identity E proportional to L^{2.57}; this identity is not established, and the variable p in [15] may not even match the parameter count in [14]. The empirical side is also fragile: four aggregate points, no error bars, no stratification details, and no seed variation. These problems are addressable with a corrected derivation and more careful scaling runs, so a conditional verdict is appropriate rather than rejection. The fine-tuning results are not the bottleneck; the scaling-law comparison is. Thus the reader's CONDITIONAL verdict remains appropriate.","tokens_in":6566,"tokens_out":5272,"duration_ms":47617,"concrete_test":"Open [15] and identify the independent variable in E~p^{-0.195}: if p denotes inference (test-time) compute rather than model parameter count, redo Eq. (1) with the correct variable and check whether the predicted d-exponent is still 0.24. Separately, rerun the Figure 2 experiment with at least 8 stratified data fractions (e.g., 100%, 75%, 50%, 33%, 25%, 17%, 12.5%, 6.25%), three random seeds per fraction, and per-task F1 values, reporting the aggregate exponent with a 95% confidence interval. If the interval excludes 0.24 by more than about +/-0.05, or if Eq. (1) cannot be repaired, the universality claim should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central scientific claim is the data-scaling exponent 0.28 matching a literature-derived 0.24 (Section IV-d). This comparison has two load-bearing gaps. First, Eq. (1) concatenates three separate empirical relations from two papers: L~p^{-0.076} and L~d^{-0.095} from [14], and E~p^{-0.195} from [15]. The algebra E~d^{0.24} only follows if (a) the p in [15] is model parameter count rather than inference-time compute, and (b) test error is a fixed power of loss throughout the relevant regime. Neither condition is shown; [15] is an inference-scaling study, so a variable mismatch is plausible. Even granting the algebra, the measured exponent 0.28 comes from Figure 2: one aggregate average-F1 curve over heterogeneous financial tasks, only four data fractions (1, 1/2, 1/4, 1/8), no seeds, no confidence intervals, and no statement that each fraction is stratified across tasks. A single outlier point could shift the fitted exponent substantially, so 'highly consistent' overstates what the data support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports supervised fine-tuning (SFT), direct preference optimization (DPO), and a pipeline here labeled reinforcement learning (RL) with model-synthesized data to adapt 1.5B-parameter open LLMs (Qwen2.5-1.5B-Instruct and DeepSeek-R1-1.5B) to financial tasks from the Open FinLLM Leaderboard. The authors report large improvements on several tasks, e.g., DeepSeek NER F1 from 0.1448 to 0.7231 after SFT, and a +87.1% relative gain on MultiFin with synthetic data. The final experimental section fits a data-scaling curve with exponent 0.28 and claims this is highly consistent with a literature-derived value of 0.24, arguing for universality of the scaling exponent across domains. The central claims are therefore empirical gains from fine-tuning and the existence of a financial-domain data scaling law; the first claim is supported by the tables, while the second rests on a fragile derivation and a four-point fit.","tokens_in":6812,"tokens_out":3508,"duration_ms":33288,"significance":"If the fine-tuning results are taken at face value, they provide useful evidence that instruction-tuned 1.5B models can be substantially improved on financial benchmarks with modest compute, and the comparison of SFT, DPO, and synthetic-data variants is a practical contribution. The paper also reports hyperparameters and training configurations, which aids reproducibility. The scaling-law claim would be significant if established, since cross-domain universality of the data-scaling exponent is an interesting hypothesis; however, as presented in Section IV-d, the derivation in Eq. (1) and the four-point aggregate fit with no error bars or seeds do not currently support the claimed agreement with 0.24. The paper's strength is the internally consistent table of fine-tuning results; its weakness is the load-bearing scaling-law comparison.","major_comments":[{"comment":"The derivation of E ~ d^{0.24} is not established. Eq. (1) combines L ~ p^{-0.076} and L ~ d^{-0.095} from [14] with E ~ p^{-0.195} from [15], which requires (a) that the variable p in [15] is model parameter count rather than inference-time compute, and (b) that test error E is a fixed power of loss L with exponent 2.57 throughout the relevant regime. Neither condition is shown, and [15] is an inference-scaling study, so the variable mismatch is plausible. The algebra also appears to drop a minus sign: if E ~ L^{2.57} and L ~ d^{-0.095}, then E ~ d^{-0.24}, not d^{0.24}; the sign convention in the comparison to the fitted exponent 0.28 should be clarified and the derivation made explicit.","section":"Section IV-d, Eq. (1)"},{"comment":"The empirical scaling exponent is fitted to only four data fractions (1, 1/2, 1/4, 1/8) with no seeds, no confidence intervals, and no statement about how data fractions were stratified across the heterogeneous financial tasks. One outlier point can materially change the fitted exponent, so the statement that 0.28 is 'very close' to 0.24 overstates the support. The authors should report per-task scaling fits, standard deviations across seeds or bootstrap intervals, and the number of tasks contributing at each fraction, or soften the universality claim accordingly.","section":"Section IV-d, Fig. 2"},{"comment":"The procedure labeled 'reinforcement learning (RL)' is actually SFT and DPO on model-synthesized data: the five steps described are corpus collection, LLM annotation via chain-of-thought, answer extraction, query formatting, and SFT/DPO training. There is no RL loss, reward model, trajectory sampling, or policy-gradient update. Calling this pipeline RL is inaccurate and affects the abstract, the introduction, and contribution 2. The authors should either rename it 'synthetic-data fine-tuning' throughout or provide an actual RL formulation. Additionally, Table III reports only relative performance boosts; absolute scores and the base-model performance for each task should be reported.","section":"Sections III-c and IV-c; Table III"}],"minor_comments":[{"comment":"The number of datasets is inconsistent: the introduction says the leaderboard contains 36 datasets over 24 tasks, while Section III-a says 'among all 41 datasets, 28 provide training/validation data.' Please reconcile the counts.","section":"Introduction and Section III-a"},{"comment":"The axis labels and fitted equation are malformed: the text shows '11/21/41/8' and 'log(1 F1) = 0.28 × log(d) 1.35', which is missing a minus sign before 1.35 and is hard to read. The variable d (data fraction) and the y-axis quantity should be printed clearly, and the sign of the exponent should be consistent with the direction of the scaling relation.","section":"Figure 2"},{"comment":"The heading 'RL with Synthesed Data' contains a typo: 'synthesed' should be 'synthesized'.","section":"Section IV-c, heading"},{"comment":"Table II shows that DPO reduces the overlength ratio and improves the unseen CC task, but the NER F1 decreases slightly; the text says this 'remains unchanged mainly' and attributes the decrease to the scoring mechanism. Please provide a more concrete explanation or report whether the difference is within evaluation noise.","section":"Section IV-b, Table II"},{"comment":"The authors use the term 'data critical exponent' for the slope of log(1-F1) versus log(data fraction). This terminology is unusual; consider using 'data-scaling exponent' and defining the exact quantity that is being fitted.","section":"Section IV-d"}],"recommendation":"major_revision","confidential_remarks":"The paper is closer to a benchmark/technical report than to a full research study, but the fine-tuning results are plausible and clearly presented. The main risk is that the scaling-law claim is the most prominent scientific contribution and it is currently supported by an unjustified cross-paper exponent composition and a very small fit. If the authors add error bars, redo the derivation carefully, and fix the RL mislabeling, the paper could be acceptable as an empirical contribution. I would also suggest the editor ensure the comparison with [15] is checked by someone familiar with inference-scaling terminology, since the variable p in that reference is likely not model parameter count."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the SFT/DPO experiments are the real content, and they are fine. The scaling-law section is the paper's claimed centerpiece, but Eq. (1) does not hold up as written, and the four-point fit without error bars cannot support the universality language.\n\nWhat is good: the transfer observations in Table I are the most interesting part. NER fine-tuning hurting CC while FiQASA fine-tuning helping FPB is the kind of concrete, task-level finding that is actually useful to practitioners. The DPO section is also well motivated — measuring the overlength ratio drop from 54.7% to 1.7% directly addresses the repetition problem, and the CC improvement is a nice side effect. The synthetic-data pipeline is a sensible recipe for leaderboard tasks without training sets, even though it is labeled \"RL\" when it is really iterative SFT/DPO on model-generated data. That naming should be fixed but is not a deep flaw.\n\nThe scaling-law claim is where I agree with the stress test. Equation (1) concatenates two Kaplan et al. exponents and one inference-scaling exponent from Wu et al. The algebra only works if the p in Wu et al. is parameter count, but that paper is about inference-time compute. Even granting that, converting test loss L to error rate E as a fixed power law across the whole regime is an assumption the paper does not justify. The fitted 0.28 comes from one aggregate F1 curve, four data fractions, no seeds, and no confidence intervals, so \"highly consistent\" overstates what the data show. The self-citation [16] for universality is not load-bearing and I would not call it a problem by itself.\n\nWho this is for: people fine-tuning 1B-scale models for finance who want a usable recipe and some honest transfer notes. The scaling-law discussion should be read skeptically until the derivation is corrected and the fit has uncertainties. The paper deserves a serious referee — a good reviewer could push the authors to fix Eq. (1), add error bars, and release code and data, which would turn a workshop paper into a citable empirical result.","headline":"A useful fine-tuning recipe for small financial LLMs with a genuinely new but fragile data-scaling exponent; the universality claim needs a corrected derivation and error bars before it can carry weight.","tokens_in":7357,"tokens_out":904,"would_cite":false,"duration_ms":9783,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning small open language models on financial benchmarks produces large task gains, and the measured financial data-scaling exponent matches the value derived from general-text and math scaling laws.","keywords":["large language models","financial NLP","supervised fine-tuning","direct preference optimization","reinforcement learning","data scaling laws","synthetic data","Open FinLLM Leaderboard"],"falsifier":"Refit the data-scaling curve separately per task with many more data fractions, say 12.5%, 6%, 3%, and 1.5%, with repeated runs to obtain error bars; if per-task exponents spread widely or the pooled exponent moves significantly away from 0.24, the universality claim is not supported.","tokens_in":6352,"feed_emoji":"📈","tokens_out":7085,"duration_ms":59522,"temperature":0.7,"pith_summary":"This paper tries to show that small, open 1.5B language models can be made strong at financial tasks through a practical fine-tuning recipe, and that financial training data follow the same scaling law seen in other domains. The authors fine-tune Qwen2.5 and DeepSeek-R1 base models with supervised fine-tuning, direct preference optimization, and a synthetic-data loop built on chain-of-thought annotation. They report large gains on Open FinLLM Leaderboard tasks, for example lifting DeepSeek's NER F1 from 0.1448 to 0.7231. They also measure a financial data-scaling exponent of 0.28 and argue it is consistent with a literature-derived exponent of 0.24, which would imply that the scaling exponent is independent of task domain. If true, this matters because it would make finance LLM development cheaper and more predictable from small-scale experiments.","feed_headline":"Fine-tuning lifts small LLMs on finance; scaling law holds","feed_subtitle":"DeepSeek NER F1 jumps from 0.14 to 0.72 and the measured data exponent 0.28 tracks the predicted 0.24.","key_machinery":"Two pieces carry the argument. The first is the fine-tuning pipeline: SFT adapts a base 1.5B model to the target format; DPO then uses the SFT model's overly long repetitive outputs as rejected examples, so the model learns to stop at the correct answer; and for datasets without training labels, the model generates its own chain-of-thought annotations, extracts answers with regular expressions, and retrains on the synthesized pairs. The second is the scaling-law identity in Eq. (1), which combines $L\\sim p^{-0.076}$, $L\\sim d^{-0.095}$, and $E\\sim p^{-0.195}$ into $E\\sim d^{0.24}$ by taking test error to be a fixed power of loss; the measured exponent 0.28 is compared with that derived value to support the universality claim.","core_discovery":"On the paper's own terms, the central discovery is that 1.5B open LLMs can be turned into competitive financial-task models with a three-stage recipe: supervised fine-tuning, direct preference optimization, and a synthetic-data loop driven by chain-of-thought annotation. SFT on the NER task raises DeepSeek-R1-1.5B's F1 from 0.1448 to 0.7231, and on FiQASA from 0.4383 to 0.7865. Applying DPO afterward cuts the overlength ratio from 54.7% to 1.7% while roughly preserving the trained-task F1 and improving the unseen CC task. For tasks with no official training split, one iteration of synthetic-data training gives a +87.1% F1 gain on MultiFin, +22.5% on FOMC, and a +3.4% RMSE improvement on TSA. The paper further claims a financial data-scaling exponent of 0.28, close to the 0.24 derived by combining prior loss and error scaling laws, and reads this agreement as evidence that the exponent is task-independent.","pith_inferences":["A per-task scaling-law fit with more data fractions and repeated runs would test whether the 0.28 exponent is a genuine universal constant or an average over heterogeneous task exponents; the paper's aggregate curve cannot distinguish these.","The synthetic-data loop is self-training, so part of its measured gain could be the model learning its own annotation biases; comparing synthetic-only training with gold-label training on tasks where gold labels exist would quantify that contamination.","Because DPO targets overly long outputs, similar fixes may be achievable at inference time through repetition penalties or sampling changes; the paper mentions this possibility in its discussion but does not test it.","If the 0.28-versus-0.24 agreement survives more careful measurement, it connects finance NLP to a physics-style critical-phenomenon picture of LLM training; that picture is an interpretation the paper offers, not something its experiments establish on their own."],"forward_implications":["Fine-tuning with LoRA on 1.5B models can reach practical score levels on specialized finance benchmarks, so strong finance LLM performance does not require large proprietary models.","DPO after SFT repairs repetitive generation and can improve an unseen classification task, making preference tuning a useful general repair step in small-model fine-tuning.","A synthetic-data loop lets teams train on leaderboard tasks that publish no training split, and even one iteration yields large gains on some tasks.","If the financial data-scaling exponent is near 0.24, practitioners can extrapolate the benefit of additional training data from a few small runs, which would make leaderboard development much cheaper.","Cross-task transfer is not uniform: fine-tuning on NER can hurt causal classification, while fine-tuning on FiQASA helps financial polarity sentiment, so the choice of fine-tuning data matters."],"supporting_citations":[{"why":"supplies the Open FinLLM Leaderboard benchmark tasks, data, and evaluation protocol that all fine-tuning results are measured on","marker":"[1]"},{"why":"contributes the Sequential Fine-Tuning and Task-Specific Prompt approach that the paper adopts and compares against","marker":"[7]"},{"why":"establishes the neural text degeneration problem that motivates the DPO stage after SFT","marker":"[10]"},{"why":"supplies learning-dynamics evidence that LLM fine-tuning produces the repetitive-output behavior fixed by DPO","marker":"[11]"},{"why":"provides the fine-tuning implementation, including LoRA, learning-rate schedule, and batch configuration, used in all SFT and DPO runs","marker":"[12]"},{"why":"provides the distributed training system used to run the fine-tuning jobs","marker":"[13]"},{"why":"gives the loss scaling laws $L\\sim p^{-0.076}$ and $L\\sim d^{-0.095}$ that the paper combines into the reference exponent","marker":"[14]"},{"why":"gives the test-error scaling $E\\sim p^{-0.195}$ used with [14] to derive the expected data exponent 0.24","marker":"[15]"},{"why":"frames the cross-domain exponent agreement as a universality phenomenon shared with phase transitions","marker":"[16]"}],"fun_headline_variants":["Scaling law holds for fine-tuned finance LLMs","DeepSeek fine-tune: NER F1 0.14 to 0.72","SFT+DPO+RL recipe lifts small LLMs on finance tasks","Finance LLM scaling exponent 0.28 matches prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is the bridge in Eq. (1): test error is a fixed power of training loss and parameter count can be exchanged for data size along a constant-loss curve; if that bridge fails, the measured exponent 0.28 cannot be compared with the derived 0.24 as evidence of universality.","fun_headline_variants_meta":{"raw":{"variants":["Scaling law holds for fine-tuned finance LLMs","DeepSeek fine-tune: NER F1 0.14 to 0.72","SFT+DPO+RL recipe lifts small LLMs on finance tasks","Finance LLM scaling exponent 0.28 matches prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3242,"prompt_tokens":895,"completion_tokens":2347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":2267}},"tokens_in":511,"tokens_out":2347,"duration_ms":16385,"temperature":1.0,"reasoning_tokens":2267,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:14:04.048321+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Refit the data-scaling curve separately per task with many more data fractions, say 12.5%, 6%, 3%, and 1.5%, with repeated runs to obtain error bars; if per-task exponents spread widely or the pooled exponent moves significantly away from 0.24, the universality claim is not supported.","supporting_citations":[{"cited_title":"FinMind-Y-me at the regulations challenge task: Financial mind your meaning based on THaLLE,","cited_arxiv_id":null,"evidence_quote":"contributes the Sequential Fine-Tuning and Task-Specific Prompt approach that the paper adopts and compares against"},{"cited_title":"Learning dynamics of llm finetuning,","cited_arxiv_id":null,"evidence_quote":"supplies learning-dynamics evidence that LLM fine-tuning produces the repetitive-output behavior fixed by DPO"},{"cited_title":"Llamafactory: Unified efficient fine-tuning of 100+ language models,","cited_arxiv_id":null,"evidence_quote":"provides the fine-tuning implementation, including LoRA, learning-rate schedule, and batch configuration, used in all SFT and DPO runs"}],"review_version":1}