{"id":"cb405d27-f391-41bc-ad40-04a67217ab0e","arxiv_id":"2509.08995","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"DPFinLLM is a standard LoRA plus DP-SGD fine-tuning recipe applied to Llama2 and ChatGLM2 for financial sentiment; the experiments are mixed, generally below state-of-the-art, and key details are missing.","lead":"This paper applies existing differential privacy and low-rank fine-tuning methods to financial sentiment models and reports accuracy on four public datasets. The reported results mostly fall short of existing FinGPT baselines, and the manuscript does not show that a new architecture or mechanism was created.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own tables contradict the 'comparable to fully fine-tuned models' claim: on TFNS, DPFinLLM is 15.8 accuracy points below FinGPT v3.2, and ChatGLM2 on NWGI matches base to five decimals.","rationale":"The reader's verdict (REJECT) is well-supported. I identify a more directly load-bearing flaw than the missing DP accounting: the utility claim fails on the paper's own reported numbers. The paper's strongest claim is about privacy plus comparable performance; the second conjunct is empirically contradicted by Tables I–IV. This makes the central claim internally inconsistent and does not depend on external reproducibility questions. The DP-accounting concern is real and secondary: §III-B gives no noise scale, lot size, sampling probabilities, or accountant, so the ε labels cannot be verified. Together these justify rejection. Because the reader's weakest assumption was about privacy accounting rather than the utility contradiction, agreement is partial rather than full. No verdict change is needed.","tokens_in":10193,"tokens_out":6082,"duration_ms":60121,"concrete_test":"Run the exact LoRA fine-tune of Llama2-7B on TFNS (same hyperparameters, data split, and training setup as §IV) with the DP noise variance set to 0 (or σ=0) and no gradient clipping, then compare to both the DPFinLLM result (0.7320) and FinGPT v3.2 (0.8899) in Table III. If the non-private LoRA control is close to 0.89, the 16-point DP drop is real and 'comparable performance' is false; if it is close to 0.73, then the baseline comparison is not apples-to-apples and the claim is still unsupported. Additionally, check the NWGI ChatGLM2 row in Table IV: recompute F1-weighted from the confusion matrix; if any metric is not above the base model, the §IV-B 'across all metrics' sentence is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; §IV-B) requires that DPFinLLM 'achieve performance comparable to fully fine-tuned models' under DP. The reported numbers already fail this on face value. On TFNS (Table III), Llama2-based DPFinLLM at ε=4 achieves 0.7320 accuracy vs FinGPT v3.2's 0.8899 — a 15.8-point gap; F1-weighted is 0.7103 vs 0.8886. On FPB (Table I), DPFinLLM is 0.7979 vs FinGPT v3.2's 0.8663. On NWGI (Table IV), the ChatGLM2-based DPFinLLM at ε=8 gives 0.56042 accuracy vs base model 0.56041 and has lower F1-weighted (0.44888 vs 0.44952), meaning the DP fine-tune did not learn on that dataset; §IV-B's statement that both models show 'substantial increases ... across all evaluation metrics' is therefore false for that row. No non-private LoRA control is reported, so the comparison cannot separate DP's utility cost from the LoRA/task setup. Even if the ε labels were real, the utility half of the headline is contradicted by the paper's own tables.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DPFinLLM, an on-device financial LLM that combines Llama2/ChatGLM2-style transformer components with LoRA fine-tuning and a DP-SGD-style privacy mechanism. It evaluates accuracy and F1 scores on four financial sentiment datasets (FPB, FIQA, TFNS, NWGI), comparing against base Llama2-7B, ChatGLM2-6B, and FinGPT baselines, and also reports zero-shot transfer results. The central claim is that DPFinLLM can safeguard user data under differential privacy while achieving performance comparable to fully fine-tuned models under strict privacy constraints.","tokens_in":10567,"tokens_out":6547,"duration_ms":77832,"significance":"If the claims were supported, the work would be relevant to privacy-preserving on-device financial sentiment analysis. The paper uses standard components (DP-SGD, LoRA, grouped-query attention, RMSNorm, SwiGLU) and does not claim new privacy theory: Theorem 1 is a restatement of the DP-SGD bound from Abadi et al. The zero-shot transfer tables are a useful attempt to study generalization under DP. However, the reported experiments do not support the headline claims: the paper's own tables show large accuracy gaps on key datasets, and the privacy accounting is not described in enough detail to verify the reported epsilon values. No reproducible artifact is actually provided; the GitHub link is unparsable and no hyperparameters are given.","major_comments":[{"comment":"The central claim that DPFinLLM 'achieve[s] performance comparable to fully fine-tuned models' is contradicted by the paper's own tables. On TFNS (Table III), Llama2-based DPFinLLM at ε=4 reaches 0.732 accuracy and 0.710 weighted F1, while FinGPT v3.2 reaches 0.890 accuracy and 0.889 weighted F1 — a 15.8-point gap. On FPB (Table I), the gap is about 6.8 points in accuracy (0.798 vs 0.866). In Table IV, the ChatGLM2-based DPFinLLM at ε=8 gives accuracy 0.56042 versus the base model's 0.56041 and a lower weighted F1 (0.44888 vs 0.44952), so the §IV-B claim of 'substantial increases ... across all evaluation metrics' is false for that row. These are not local presentation issues; they undermine the abstract's and conclusion's main assertion.","section":"Abstract; §IV-B; Tables I, III, IV"},{"comment":"The privacy guarantee is not instantiated. Theorem 1 is quoted with unspecified constants c1 and c2, and the experimental section does not report the noise scale σ, gradient clipping bound C, lot size L, sampling probability q, number of training steps T, or the privacy accountant used to convert these into ε. The paper sets δ=1/|d| but gives no values for the other quantities. In DP-SGD, ε is the output of a privacy accountant given those hyperparameters; simply selecting ε and invoking Eq. (11) is not a reproducible privacy argument. This is load-bearing for the paper's 'safeguard user data' claim.","section":"§III-B, Eq. (11); §IV-A5"},{"comment":"There is no non-private LoRA control. DPFinLLM is fine-tuned with LoRA, but the comparisons are against base models and FinGPT baselines, which are described as fully fine-tuned. Without a same-architecture LoRA fine-tune without DP, the reported accuracy differences cannot be attributed to the DP mechanism rather than to LoRA or the training setup. This is necessary to support the conclusion that DPFinLLM is comparable to fully fine-tuned models 'even under strict privacy constraints'.","section":"§IV-A2, §IV-B"},{"comment":"All results are single-run point estimates with no standard deviations, confidence intervals, or number of seeds. Several differences are small enough to be noise (e.g., FIQA ChatGLM2-based DPFinLLM accuracy 0.83636 versus ChatGLM2-6B 0.83636; NWGI ChatGLM2 row 0.56042 versus 0.56041). The empirical claims of 'substantial increases' therefore cannot be robustly assessed from the reported data.","section":"§IV-A5, Tables I–IV"}],"minor_comments":[{"comment":"Stray comma in the clipping formula: 'max(1, ,||g_t(x_i)||_2/C)' should read 'max(1, ||g_t(x_i)||_2/C)'.","section":"Eq. (7)"},{"comment":"The differential privacy inequality is malformed. It should be Pr[M(d) ∈ S] ≤ e^ε Pr[M(d') ∈ S] + δ, with set membership and proper exponent formatting.","section":"Eq. (10)"},{"comment":"The sentence comparing against 'the original Llama2-7B and ChatGLM2-6B models, both trained on a financial multi-task dataset' conflates base pretrained models with FinGPT, which is the model trained on the financial multitask dataset.","section":"§IV-A2"},{"comment":"The open-source code URL contains a line break and a space ('DP FinLLM'), so it is not usable as given.","section":"§IV, github link"},{"comment":"The figure caption and text do not state which base model or dataset is shown, nor how the other hyperparameters are fixed when ε is varied. The figure is not self-contained.","section":"Fig. 1"}],"recommendation":"reject","confidential_remarks":"For the editor: the paper's central empirical claim is directly contradicted by its own reported numbers, and the privacy accounting is not reproducible. I do not see a path to a minor revision; a major rework with non-private controls, proper privacy accounting, repeated runs, and a more modest claim would be needed before the work could be considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper's central selling point doesn't survive contact with its own tables. The abstract and Section IV-B claim DPFinLLM performs comparably to fully fine-tuned models under strict privacy. On TFNS, Llama2-based DPFinLLM at epsilon=4 gets 0.7320 accuracy while FinGPT v3.2 gets 0.8899, a 15.8-point gap. On FPB the gap is about 7 points. And on NWGI, ChatGLM2-based DPFinLLM at epsilon=8 scores 0.56042 accuracy versus base model 0.56041, with F1-weighted actually lower. If that's not a copy-paste error, the fine-tune learned nothing, and Section IV-B's 'substantial increases across all evaluation metrics' is false for that row.\n\nWhat's genuinely here: it's a clean application of DP-SGD (Abadi et al.) with LoRA to four public financial sentiment datasets. The math in Section III is standard — restating known definitions and quoting Theorem 1 from Abadi. The zero-shot tables (V and VI) are a reasonable descriptive addition, even if they lack statistical grounding. If the hyperparameters and accountant were given, this could be a useful reproducibility data point.\n\nBut the soft spots are not minor. There is no non-private LoRA baseline, so you can't separate the privacy cost from the LoRA/task setup. No error bars. No noise scale, clipping bound, lot size, sampling scheme, number of steps, or accountant name, making the 'epsilon=8' labels unverifiable. The GitHub URL in Section IV contains a space ('DP FinLLM') and appears non-functional. The on-device deployment framing is asserted, not backed by any system or latency evidence.\n\nBottom line: this is a parameter scan of known methods wrapped in overclaiming. The internal contradiction with the NWGI row would need to be resolved before any serious review. I would desk reject it; the authors could salvage a short empirical note if they fix the overclaims and provide the missing details, but as it stands I wouldn't cite it or spend a reading group on it unless you want a case study in how not to report DP fine-tuning results.","headline":"Routine DP-SGD + LoRA application whose own tables refute the 'comparable to fully fine-tuned models' claim; the ChatGLM2 NWGI row matches the base model to the fifth decimal.","tokens_in":11065,"tokens_out":3324,"would_cite":false,"duration_ms":32600,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a privacy-protecting, lightweight financial LLM can be fine-tuned with differential privacy and still match fully fine-tuned models on financial sentiment at strict privacy budgets.","keywords":["differential privacy","financial sentiment analysis","large language models","low-rank adaptation","on-device AI","privacy-preserving machine learning","gradient clipping","Gaussian noise"],"falsifier":"Recompute the privacy budget with a standard accountant (e.g., Rényi DP) using the training configuration from the released code; if the realized epsilon exceeds the reported values, the privacy claim is false. Alternatively, retrain the fully fine-tuned baselines on exactly the same per-dataset training subsets; if their accuracy then matches or exceeds the DP models, the utility claim fails.","tokens_in":10088,"feed_emoji":"🔐","tokens_out":5663,"duration_ms":56354,"temperature":0.7,"pith_summary":"The paper sets out to show that privacy and accuracy can coexist in financial LLMs deployed on edge devices. It introduces DPFinLLM, a lightweight transformer fine-tuned with LoRA under differential privacy, and reports that on four financial sentiment datasets it achieves accuracy and F1 scores comparable to fully fine-tuned models, with privacy budgets as strict as epsilon=2. This matters because on-device financial tools handle sensitive data, and differential privacy is the standard formal defense against membership and model-inversion attacks. The paper also reports that DP fine-tuning preserves most zero-shot generalization to unseen financial datasets, with one notable cross-dataset drop. The practical value hinges on whether the reported epsilon bounds are real, which the paper does not fully document.","feed_headline":"Differential privacy matches full fine-tuning on financial sentiment","feed_subtitle":"On-device financial assistants could protect user data without sacrificing sentiment accuracy, if reported privacy budgets hold.","key_machinery":"The load-bearing mechanism is the differential-private fine-tuning loop: per-sample gradient clipping with norm bound C (Eq. 7), averaging over a lot, and Gaussian noise injection scaled by sigma (Eq. 8), whose privacy cost is bounded by Theorem 1 in the style of DPSGD. The rank-restricted LoRA update (Eq. 4) is the companion mechanism that makes this affordable for large base models. Together they let the paper claim both privacy and task performance, since the noise is added to low-rank adapters rather than the full model.","core_discovery":"The paper claims that DPFinLLM, a lightweight transformer fine-tuned with LoRA under an (epsilon, delta)-differential privacy mechanism, performs on par with fully fine-tuned financial sentiment models. Using Llama2-7B and ChatGLM2-6B as base architectures, the pipeline clips per-example gradients and adds Gaussian noise to batch gradients, following the standard DPSGD recipe. Across FPB, FIQA, TFNS, and NWGI, the DP-fine-tuned models improve over the base models and, at epsilon values from 2 to 8, reach accuracy and F1 scores comparable to the best fully fine-tuned baselines, with zero-shot transfer to unseen datasets mostly preserved. The authors interpret this as evidence that privacy-pro","pith_inferences":["We infer that the reported privacy guarantees are not yet independently verifiable: the paper omits the noise scale, clipping bound, lot size, sampling scheme, training steps, and the privacy accountant, so a reader cannot confirm that the reported epsilon values bound the actual runs.","We infer that the comparison to fully fine-tuned baselines is potentially confounded, since the DP models are fine-tuned per dataset while the strongest baselines are trained on a multi-task corpus; a clean test would match training sets and data order.","We infer that the architecture choices (grouped-query attention, RMSNorm, SwiGLU, rotary embeddings) are not themselves evaluated; a reader cannot tell whether the privacy-utility result depends on these choices or would transfer to any transformer base.","We infer that the same pipeline should be tested on non-sentiment financial tasks (e.g., named-entity recognition, fraud classification) before claiming general on-device financial applicability."],"forward_implications":["If the reported results hold, on-device financial sentiment models can be fine-tuned with a formal privacy guarantee at a small accuracy cost, enabling use on user-facing devices.","The non-monotonic relationship between epsilon and F1 across datasets implies that privacy budgets should be tuned per dataset rather than simply set to the largest acceptable value.","The zero-shot results suggest DP fine-tuning can preserve generalization to unseen financial text, so a single privately fine-tuned model could be deployed across related sentiment tasks.","Because the method uses LoRA, the privacy mechanism adds only small memory and compute overhead, making it viable on resource-constrained hardware."],"fun_headline_variants":["DP fine-tuning matches full fine-tune on financial sentiment","On-device financial LLMs: DP matches no-privacy accuracy","Privacy-preserving financial LLMs match full fine-tune accuracy","DPFinLLM: Private fine-tuning equals non-private on sentiment","Differential privacy: financial sentiment accuracy without leaks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim stands or falls on whether the reported epsilon values truly bound the privacy loss of the actual fine-tuning runs, yet the paper never discloses the noise scale, clipping bound, lot size, sampling scheme, or privacy accountant used.","fun_headline_variants_meta":{"raw":{"variants":["DP fine-tuning matches full fine-tune on financial sentiment","On-device financial LLMs: DP matches no-privacy accuracy","Privacy-preserving financial LLMs match full fine-tune accuracy","DPFinLLM: Private fine-tuning equals non-private on sentiment","Differential privacy: financial sentiment accuracy without leaks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1343,"prompt_tokens":702,"completion_tokens":641,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":446,"tokens_out":641,"duration_ms":6814,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:50:44.765990+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the privacy budget with a standard accountant (e.g., Rényi DP) using the training configuration from the released code; if the realized epsilon exceeds the reported values, the privacy claim is false. Alternatively, retrain the fully fine-tuned baselines on exactly the same per-dataset training subsets; if their accuracy then matches or exceeds the DP models, the utility claim fails.","supporting_citations":[],"review_version":1}