REVIEW 4 major objections 5 minor 1 cited by
FinLoRA: Finetuning Quantized Financial Large Language Models Using Low-Rank Adaptation
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Finetuning Llama 3.1 models with quantized low-rank adaptation lifts accuracy on financial tasks while cutting GPU memory enough to run on a single 48GB GPU.
desk verdict Solid workshop application paper with useful memory numbers and a new XBRL extraction set, but the loose evaluation protocol keeps the 48% accuracy claim from being fully supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the low-rank update $\Delta W = BA$ added to frozen pretrained weights $W_0$, making the forward pass $y = W_0 x + BA x$ with rank $r \ll n$; QLoRA keeps $W_0$ in 4-bit or 8-bit quantized form and leaves only the small adapter matrices $A$ and $B$ in 16-bit precision. The low rank shrinks trainable parameters to about 0.039% of the original weight matrix at $n=4096$, $r=8$, and quantization shrinks resident model size; the paper adds distributed data parallelism, pipeline parallelism, BF16, and a communication-efficient optimizer to make local finetuning and inference faster and cheaper.
What would settle it
Evaluate the base Llama 3.1 8B Instruct on FPB and NER using exactly the instruction prompts, one-shot examples, and inclusion-based scoring given to the finetuned models; if its 68.73% and 48.89% scores jump toward the finetuned 86.30% and 96.63%, the claimed accuracy gains are mostly prompt-format compliance rather than learned financial skill.
Extended reading notes
Core claim
The central discovery is empirical: quantized low-rank finetuning turns an instruction-tuned general model into a strong financial model across sentiment analysis, named entity recognition, headline classification, and XBRL tagging and extraction. With a rank-4 LoRA adapter on a 4-bit quantized Llama 3.1 8B, accuracy rises from 68.73% to 86.30% on FPB, from 48.89% to 96.63% on NER, from 45.34% to 88.03% on Headline, and from 2.85% to 70.45% on FiNER, while the 70B quantized-finetuned model reaches 80.94%, 98.88%, and 96.38% on FPB, NER, and Headline. The same configuration cuts GPU memory for 8B finetuning from 30.9 GB at 16-bit to 8.7 GB at 4-bit and for 70B from over 300 GB to 42.8 GB, and the paper states all finetuning can run on a single 48GB GPU with longer training time.
Load-bearing premise
The load-bearing premise is that the base models received the same task instructions, one-shot examples, and lenient substring-inclusion scoring as the finetuned models, since otherwise the reported accuracy gaps could reflect output-format compliance rather than financial knowledge learned by finetuning.
Editorial extensions
If this is right
- A 4-bit rank-4 adapter is enough for the 8B model to reach 86.30% on FPB and 96.63% on NER, so memory-saving settings do not cost accuracy.
- The QLoRA-finetuned 8B model outscores the unadapted 70B base model on all nine reported tasks, showing task adaptation can beat raw scale for financial workloads.
- Finetuning the 70B model in 4-bit fits in 42.8 GB of GPU memory, which puts large-model financial finetuning within reach of a single high-end GPU.
- XBRL tagging accuracy jumps from 2.85% to 70.45% and XBRL tag/value extraction from 79.37%/55.26% to 95.00%/96.05%, making local LLM-based XBRL report analysis practical.
Reading between the lines
- Inference: Because scoring is inclusion-based, a strict exact-match rescoring of the same outputs would separate real gains in factual precision from gains in producing verbose text that happens to contain the answer.
- Inference: The paper leaves the 70B QLoRA model unevaluated on XBRL tasks; running it on FiNER, tag, and value extraction would show whether the 8B XBRL gains transfer to the larger model.
- Inference: The FPB train/test split is manually created; repeating the sentiment experiments on the standard public split would indicate how much of the 68.73% to 86.30% jump depends on split choice.
- Inference: Keeping adapter weights separate at inference invites a multi-adapter router that lets one quantized base model switch among sentiment, NER, headline, and XBRL tasks, which the paper names as its future mixture-of-experts direction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FinLoRA, a recipe for locally finetuning quantized large language models (Llama 3.1 8B and 70B Instruct) on financial tasks using QLoRA, distributed data parallelism, and pipeline parallelism. The authors report accuracy and F1 results on six classification-style datasets (FPB, FiQA SA, TFNS, NWGI, Headline, NER) and three XBRL tasks (Tagging, Tags, Values), along with GPU memory, training time, adapter size, and inference speed measurements. The central empirical claim is that QLoRA finetuning yields large accuracy improvements over the base models (e.g., from 48.89% to 96.63% on NER for the 8B 4-bit rank-4 model) while reducing GPU memory requirements and enabling finetuning on commodity GPUs.
Significance. If the evaluation is fair, the paper provides a practically useful demonstration that QLoRA can adapt publicly available instruction-tuned LLMs to diverse financial tasks with modest compute, and the resource-usage tables are a useful reference for practitioners. The manuscript's strengths include the use of public datasets, a released code repository, and a clear presentation of the quantization and parallelism techniques. The headline accuracy gains are plausible in direction, but the current evidence does not yet isolate the source of those gains: the base-model evaluation protocol is underspecified, the accuracy metric is permissive, and the construction of the FPB split is not documented. These are fixable concerns, so the contribution could become acceptable after a revised evaluation that controls for prompt, decoding, and scoring differences between base and finetuned models.
major comments (4)
- [Implementation Details / Performance Metrics] The central claim that QLoRA finetuning yields large accuracy gains over the base Llama models requires that base and finetuned models are evaluated under identical conditions. The paper does not report the exact instruction prompts, few-shot templates, decoding parameters, or answer parsers used for the base models on the classification tasks; the only mention of task-specific prompting is the one-shot example for XBRL extraction 'to allow better instruction following for the base model' (Implementation Details). If the base models were queried without task-specific instructions or with different output-format expectations, a substantial part of the reported gains could reflect format compliance rather than acquired financial knowledge. Please provide the full prompt templates used for both base and finetuned models, the generation settings (temperature, max tokens, etc.), and the programmatic scoring procedure, and rerun the base-model evaluations under those same conditions.
- [Performance Metrics] The accuracy metric is defined as 'an answer is considered correct if the ground truth answer is included in the generated response.' This inclusion test is too lenient for the classification and extraction tasks: a model that emits multiple candidate labels or verbose text containing the correct token can be scored correct even when it has not isolated the intended answer. This is especially consequential for NER (where 'location' might appear incidentally) and for XBRL value extraction (where a number might appear in a longer narrative). Please report exact-match accuracy or a stricter structured-output metric, and describe the parsing used to extract the answer from the generated text. If the lenient metric is retained, justify its validity for each task.
- [Abstract / Conclusion / Table 3] The statement 'up to a 48% average increase in accuracy' is not backed by a clear computation. For Llama-3.1-8B-4bits-r4 relative to the 8B base, the per-task relative improvements in Table 3 range roughly from 19.7% (XBRL Tags) to 97.7% (NER), with the FiNER value an extreme outlier (2.85% to 70.45%); the simple average across the nine tasks is well above 48%, while the average across the six classification tasks is near 62%, and for the 70B model the average is nearer 33%. Please specify exactly which tasks and which averaging scheme produce the claimed 48%, and ensure the wording distinguishes 'up to 48% average increase' from the per-task maxima shown in the table.
- [Datasets] The FPB train/test split is described only as 'manually created' with train/test sizes of 1.2K/3.6K. This is not reproducible and creates a risk of distribution shift or leakage relative to the standard FPB split used in prior work. Please provide the exact split (e.g., as a data file or hash of the indices) or use a publicly available split, and state whether any hyperparameter or early-stopping decisions were made using the test set.
minor comments (5)
- [Abstract] The abstract promises improvements in 'accuracy, GPU memory usage, and time efficiency,' but time-to-accuracy or training-time comparisons against a non-quantized baseline are not provided; consider qualifying this claim.
- [Table 3] The column headers 'Tags V alues' contain a spacing artifact; also, F1 scores are only reported for the classification datasets, not for the XBRL tasks, and the dashes make the table harder to read. Consider reporting complete metric rows or stating why F1 is omitted for XBRL.
- [Finetuning LLMs with Quantized Low-rank Adaptation (QLoRA)] The sentence 'This is different from (Hu et al. 2021), because we will explore the Mixture of Experts approach...' is a forward-looking justification that is not substantiated by any experiments in this paper; it should be rewritten to describe the actual inference behavior used in the reported results.
- [Implementation Details] Please specify the number of training runs, seeds, and variance across runs; no standard deviations or error bars are reported for any of the accuracy or memory numbers.
- [Global] There are several typos, including 'consistancy' in the Datasets section, 'show casing' in Results and Analysis, and inconsistent spacing in references; a careful proofread is needed.
Circularity Check
No significant circularity: the accuracy claims are empirical comparisons against base Llama models on public datasets, with no derivation that reduces to its own inputs.
full rationale
The paper's central claim is an empirical one: QLoRA finetuning improves accuracy on financial tasks relative to the base Llama 3.1 Instruct models. The reported numbers come from evaluations on public datasets (FPB, FiQA, TFNS, NWGI, Headline, NER, FiNER, and XBRL extraction tasks), not from any equation that fits the outcome by construction. The method section restates standard LoRA/QLoRA definitions and engineering choices; no 'prediction' is derived from a parameter fitted to the same target quantity. The paper does cite prior work by overlapping authors (FinGPT, Liu et al. 2024a, 0/1 Adam, XBRL Agent), but these citations support background, optimizer choice, and dataset provenance rather than the central accuracy result. No uniqueness theorem or ansatz is imported from the authors' prior work to force the conclusion. The main evaluative weakness is that the exact prompts and decoding settings for the base models on classification tasks are not reported, and the inclusion-based accuracy metric may reward verbose outputs; however, this is an evaluation-protocol concern, not a circularity where an output equals its input by definition. The measured comparisons are externally grounded against base models on public benchmarks, so the circularity burden is low.
Assumptions & free parameters
free parameters (6)
- LoRA rank (r) =
4, 8
- Learning rate =
1e-4
- LoRA alpha =
32
- LoRA dropout =
0.1
- Number of epochs =
4 for classification, 1 for XBRL extraction
- FPB train/test split =
1.2K/3.6K
assumptions (4)
- domain assumption QLoRA reduces memory usage while maintaining model performance
- ad hoc to paper The accuracy metric (ground truth string appears in the generated response) is a valid measure of task performance
- domain assumption The public datasets and manual FPB split are correctly labeled and representative
- domain assumption Llama 3.1 models are appropriate base models for financial tasks
Cite this review
Pith. "Pith review of FinLoRA: Finetuning Quantized Financial Large Language Models Using Low-Rank Adaptation." pith.science (2026). https://pith.science/paper/LPIBA6DG
@misc{pith2026241211378,
author = {Pith},
title = {Pith review of: FinLoRA: Finetuning Quantized Financial Large Language Models Using Low-Rank Adaptation},
year = {2026},
howpublished = {\url{https://pith.science/paper/LPIBA6DG}},
note = {Machine review of arXiv:2412.11378}
}
read the original abstract
Finetuned large language models (LLMs) have shown remarkable performance in financial tasks, such as sentiment analysis and information retrieval. Due to privacy concerns, finetuning and deploying Financial LLMs (FinLLMs) locally are crucial for institutions. However, finetuning FinLLMs poses challenges including GPU memory constraints and long input sequences. In this paper, we employ quantized low-rank adaptation (QLoRA) to finetune FinLLMs, which leverage low-rank matrix decomposition and quantization techniques to significantly reduce computational requirements while maintaining high model performance. We also employ data and pipeline parallelism to enable local finetuning using cost-effective, widely accessible GPUs. Experiments on financial datasets demonstrate that our method achieves substantial improvements in accuracy, GPU memory usage, and time efficiency, underscoring the potential of lowrank methods for scalable and resource-efficient LLM finetuning.
Forward citations
Cited by 1 Pith paper
-
Open FinLLM Leaderboard: Towards Financial AI Readiness
The paper presents an open, continuously updated FinLLM leaderboard that aggregates existing financial benchmarks and demos for comparing models.
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Chen, J.; Hu, Y.; Wang, Y.; Lu, Y.; Cao, X.; Lin, M.; Xu, H.; Wu, J.; Xiao, C.; Sun, J.; et al. 2024 a . Trialbench: Multi-modal artificial intelligence-ready clinical trial datasets. arXiv preprint arXiv:2407.00631
arXiv 2024
-
[4]
E.; Herrington, D
Chen, L.; Lu, Y.; Wu, C.-T.; Clarke, R.; Yu, G.; Van Eyk, J. E.; Herrington, D. M.; and Wang, Y. 2021. Data-driven detection of subtype-specific differentially expressed genes. Scientific reports, 11(1): 332
2021
-
[5]
Chen, T.; Hao, N.; Lu, Y.; and Van Rechem, C. 2024 b . Uncertainty Quantification on Clinical Trial Outcome Prediction. arXiv preprint arXiv:2401.03482
arXiv 2024
-
[6]
Chen, T.; Hao, N.; Van Rechem, C.; Chen, J.; and Fu, T. 2024 c . Uncertainty quantification and interpretability for clinical trial approval prediction. Health Data Science, 4: 0126
work page 2024
-
[7]
Dettmers, T.; Pagnoni, A.; Holtzman, A.; and Zettlemoyer, L. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314
arXiv 2023
-
[8]
Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; and et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783
arXiv 2024
Show all 27 references
-
[9]
M.; and Wang, Y
Fu, Y.; Lu, Y.; Wang, Y.; Zhang, B.; Zhang, Z.; Yu, G.; Liu, C.; Clarke, R.; Herrington, D. M.; and Wang, Y. 2024. DDN3. 0: Determining significant rewiring of biological network structure with differential dependency networks. Bioinformatics, btae376
2024
-
[10]
Han, S.; Kang, H.; Jin, B.; Liu, X.-Y.; and Yang, S. Y. 2024. XBRL Agent: Leveraging Large Language Models for Financial Report Analysis. In Proceedings of the 5th ACM International Conference on AI in Finance, ICAIF '24, 856–864. New York, NY, USA: Association for Computing M...
2024
-
[11]
J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W
Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685
2021 arXiv
-
[12]
D.; Mudigere, D.; Mellempudi, N.; Das, D.; Banerjee, K.; Avancha, S.; Vooturi, D
Kalamkar, D. D.; Mudigere, D.; Mellempudi, N.; Das, D.; Banerjee, K.; Avancha, S.; Vooturi, D. T.; Jammalamadaka, N.; Huang, J.; Yuen, H.; Yang, J.; Park, J.; Heinecke, A.; Georganas, E.; Srinivasan, S. M.; Kundu, A.; Smelyanskiy, M.; Kaul, B.; and Dubey, P. K. 2019. A Study o...
2019 arXiv
-
[13]
Li, S.; Zhao, Y.; Varma, R.; Salpekar, O.; Noordhuis, P.; Li, T.; Paszke, A.; Smith, J.; Vaughan, B.; Damania, P.; and Chintala, S. 2020. PyTorch distributed: experiences on accelerating data parallel training. Proc. VLDB Endow., 13(12): 3005–3018
2020
-
[14]
Liu, X.-Y.; Wang, G.; Yang, H.; and Zha, . D. 2023. Data-centric FinGPT : Democratizing Internet-scale data for financial large language models. In Workshop on Instruction Tuning and Instruction Following, NeurIPS
2023
-
[15]
Liu, X.-Y.; Zhang, J.; Wang, G.; Tong, W.; and Walid, A. 2024 a . Efficient Pretraining and Finetuning of Quantized LLMs with Low-Rank Structure . In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS), 300--311. Los Alamitos, CA, USA: IEEE Compute...
2024
-
[16]
Liu, X.-Y.; Zhu, R.; Zha, D.; Gao, J.; Zhong, S.; White, M.; and Qiu, M. 2024 b . Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning. ACM Transactions on Management Information Systems
2024
-
[17]
Loukas, L.; Fergadiotis, M.; Chalkidis, I.; Spyropoulou, E.; Malakasiotis, P.; Androutsopoulos, I.; and Paliouras, G. 2022. F i NER : Financial Numeric Entity Recognition for XBRL Tagging. In Muresan, S.; Nakov, P.; and Villavicencio, A., eds., Proceedings of the 60th Annual M...
2022
-
[18]
D.; and He, Y
Lu, Y.; Li, C.; Zhang, M.; Sa, C. D.; and He, Y. 2022 a . Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam. arXiv:2202.06009
2022 arXiv
-
[19]
J.; Cheng, Z.; Saylor, G.; Van Eyk, J
Lu, Y.; Wu, C.-T.; Parker, S. J.; Cheng, Z.; Saylor, G.; Van Eyk, J. E.; Yu, G.; Clarke, R.; Herrington, D. M.; and Wang, Y. 2022 b . COT : an efficient and accurate method for detecting marker genes among many subtypes. Bioinformatics Advances, 2(1): vbac037
2022
-
[20]
Maia, M.; Handschuh, S.; Freitas, A.; Davis, B.; McDermott, R.; Zarrouk, M.; and Balahur, A. 2018. WWW'18 Open Challenge: Financial Opinion Mining and Question Answering. 1941--1942
2018
-
[21]
Malo, P.; Sinha, A.; Takala, P.; Korhonen, P.; and Wallenius, J. 2013. Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts. arXiv:1307.5336
2013 arXiv
-
[22]
Rahman, M. A. 2022. Twitter financial news sentiment. http://precog.iiitd.edu.in/people/anupama
2022
-
[23]
Saeedi, A.; Richards, J.; and Smith, B. 2007. An Introduction to XBRL. In British Accounting Association s Annual Conference
2007
-
[24]
C.; Verspoor, K.; and Baldwin, T
Salinas Alvarado, J. C.; Verspoor, K.; and Baldwin, T. 2015. Domain Adaption of Named Entity Recognition to Support Credit Risk Assessment. In Hachey, B.; and Webster, K., eds., Proceedings of the Australasian Language Technology Association Workshop 2015, 84--90. Parramatta, ...
2015
-
[25]
Sinha, A.; and Khandait, T. 2020. Impact of News on the Commodity Market: Dataset and Results. arXiv:2009.04202
2020 arXiv
-
[26]
S.; Zha, D.; White, M.; Xiao, K.; and Liu, X.-Y
Tian, F.; Byadgi, A.; Kim, D. S.; Zha, D.; White, M.; Xiao, K.; and Liu, X.-Y. 2024. Customized FinGPT Search Agents Using Foundation Models. In Proceedings of the 5th ACM International Conference on AI in Finance, 469--477
2024
-
[27]
Wang, Y.; Xu, Y.; Ma, Z.; Xu, H.; Du, B.; Gao, H.; and Wu, J. 2024. TWIN-GPT : Digital Twins for Clinical Trials via Large Language Model. arXiv preprint arXiv:2404.01273
2024 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.