{"id":"64c52429-7324-450e-88fa-a35dca014847","arxiv_id":"2412.11378","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Finetuning Llama 3.1 8B/70B with QLoRA on financial datasets improves accuracy over base models while reducing GPU memory, though the evaluation lacks baselines and the headline gain is imprecise.","lead":"This paper applies QLoRA (quantized low-rank adaptation) to finetune Llama 3.1 models on financial tasks like sentiment analysis and XBRL extraction, reporting large accuracy gains over base models with lower GPU memory use. It shows that local, resource-efficient finetuning of financial LLMs is practical, but the evaluation compares only against base models and uses a lenient metric.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main accuracy claim rests on an uncontrolled evaluation protocol: base models may receive different prompts or scoring than QLoRA-finetuned models, so the reported gains are not yet identified as task competence rather than format compliance.","rationale":"The reader correctly identified the most load-bearing assumption: the evaluation protocol must be fair and symmetric across base and finetuned models. My review confirms that the manuscript does not document the base-model prompts for the classification tasks, does not specify decoding parameters, and uses a lenient inclusion metric. These omissions make the central accuracy claim underdetermined. I agree with the reader's conditional assessment: the result is plausible and consistent with prior QLoRA/FinGPT work, but the evidence as reported is insufficient for an unconditional accept. I do not find a more severe internal inconsistency or a reason to reject the paper outright, provided the requested protocol details and matched baselines are supplied. The memory-efficiency claims, while reported on slightly different bases across tables, are broadly consistent with per-GPU versus multi-GPU aggregation and do not threaten the central claim as much as the uncontrolled evaluation does.","tokens_in":8243,"tokens_out":5954,"duration_ms":56835,"concrete_test":"Rerun the Table 3 comparison with the base Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct models using the exact same task-specific instruction prompt, output-format guidance, and one-shot example (for XBRL tasks) as used for the finetuned models, with the same decoding settings and the same inclusion-based scorer. Also compute exact-match accuracy for both base and finetuned models. If the base model accuracy rises substantially under the matched prompt, or if the finetuned-model accuracy drops substantially under exact-match scoring, the reported QLoRA gains are largely an artifact of protocol differences; if both numbers remain close to Table 3, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that QLoRA finetuning produces substantial accuracy gains over base Llama 3.1 Instruct models. For this claim to hold, the base and finetuned models must be evaluated under identical conditions: the same task instructions, the same answer format constraints, the same decoding settings, and the same scoring rule. The paper does not specify the prompts used for the base models on the classification tasks (FPB, FIQA, TFNS, NWGI, Headline, NER). It only states that one-shot prompting is used for XBRL extraction 'to allow better instruction following for the base model' (Implementation Details). If the base models were queried without task-specific instructions or with a different output format, a large fraction of the accuracy gap could reflect the finetuned model learning to emit the expected label format rather than acquiring financial knowledge. The scoring rule compounds this risk: 'an answer is considered correct if the ground truth answer is included in the generated response' (Performance Metrics). Under inclusion scoring, a model that produces a verbose or multi-candidate answer may be scored correct even when it has not precisely identified the target. Without reporting the exact prompt template, decoding parameters, and answer parser applied to both base and finetuned models, the headline accuracy improvements are not uniquely attributable to QLoRA finetuning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FinLoRA, a recipe for locally finetuning quantized large language models (Llama 3.1 8B and 70B Instruct) on financial tasks using QLoRA, distributed data parallelism, and pipeline parallelism. The authors report accuracy and F1 results on six classification-style datasets (FPB, FiQA SA, TFNS, NWGI, Headline, NER) and three XBRL tasks (Tagging, Tags, Values), along with GPU memory, training time, adapter size, and inference speed measurements. The central empirical claim is that QLoRA finetuning yields large accuracy improvements over the base models (e.g., from 48.89% to 96.63% on NER for the 8B 4-bit rank-4 model) while reducing GPU memory requirements and enabling finetuning on commodity GPUs.","tokens_in":8457,"tokens_out":3152,"duration_ms":29300,"significance":"If the evaluation is fair, the paper provides a practically useful demonstration that QLoRA can adapt publicly available instruction-tuned LLMs to diverse financial tasks with modest compute, and the resource-usage tables are a useful reference for practitioners. The manuscript's strengths include the use of public datasets, a released code repository, and a clear presentation of the quantization and parallelism techniques. The headline accuracy gains are plausible in direction, but the current evidence does not yet isolate the source of those gains: the base-model evaluation protocol is underspecified, the accuracy metric is permissive, and the construction of the FPB split is not documented. These are fixable concerns, so the contribution could become acceptable after a revised evaluation that controls for prompt, decoding, and scoring differences between base and finetuned models.","major_comments":[{"comment":"The central claim that QLoRA finetuning yields large accuracy gains over the base Llama models requires that base and finetuned models are evaluated under identical conditions. The paper does not report the exact instruction prompts, few-shot templates, decoding parameters, or answer parsers used for the base models on the classification tasks; the only mention of task-specific prompting is the one-shot example for XBRL extraction 'to allow better instruction following for the base model' (Implementation Details). If the base models were queried without task-specific instructions or with different output-format expectations, a substantial part of the reported gains could reflect format compliance rather than acquired financial knowledge. Please provide the full prompt templates used for both base and finetuned models, the generation settings (temperature, max tokens, etc.), and the programmatic scoring procedure, and rerun the base-model evaluations under those same conditions.","section":"Implementation Details / Performance Metrics"},{"comment":"The accuracy metric is defined as 'an answer is considered correct if the ground truth answer is included in the generated response.' This inclusion test is too lenient for the classification and extraction tasks: a model that emits multiple candidate labels or verbose text containing the correct token can be scored correct even when it has not isolated the intended answer. This is especially consequential for NER (where 'location' might appear incidentally) and for XBRL value extraction (where a number might appear in a longer narrative). Please report exact-match accuracy or a stricter structured-output metric, and describe the parsing used to extract the answer from the generated text. If the lenient metric is retained, justify its validity for each task.","section":"Performance Metrics"},{"comment":"The statement 'up to a 48% average increase in accuracy' is not backed by a clear computation. For Llama-3.1-8B-4bits-r4 relative to the 8B base, the per-task relative improvements in Table 3 range roughly from 19.7% (XBRL Tags) to 97.7% (NER), with the FiNER value an extreme outlier (2.85% to 70.45%); the simple average across the nine tasks is well above 48%, while the average across the six classification tasks is near 62%, and for the 70B model the average is nearer 33%. Please specify exactly which tasks and which averaging scheme produce the claimed 48%, and ensure the wording distinguishes 'up to 48% average increase' from the per-task maxima shown in the table.","section":"Abstract / Conclusion / Table 3"},{"comment":"The FPB train/test split is described only as 'manually created' with train/test sizes of 1.2K/3.6K. This is not reproducible and creates a risk of distribution shift or leakage relative to the standard FPB split used in prior work. Please provide the exact split (e.g., as a data file or hash of the indices) or use a publicly available split, and state whether any hyperparameter or early-stopping decisions were made using the test set.","section":"Datasets"}],"minor_comments":[{"comment":"The abstract promises improvements in 'accuracy, GPU memory usage, and time efficiency,' but time-to-accuracy or training-time comparisons against a non-quantized baseline are not provided; consider qualifying this claim.","section":"Abstract"},{"comment":"The column headers 'Tags V alues' contain a spacing artifact; also, F1 scores are only reported for the classification datasets, not for the XBRL tasks, and the dashes make the table harder to read. Consider reporting complete metric rows or stating why F1 is omitted for XBRL.","section":"Table 3"},{"comment":"The sentence 'This is different from (Hu et al. 2021), because we will explore the Mixture of Experts approach...' is a forward-looking justification that is not substantiated by any experiments in this paper; it should be rewritten to describe the actual inference behavior used in the reported results.","section":"Finetuning LLMs with Quantized Low-rank Adaptation (QLoRA)"},{"comment":"Please specify the number of training runs, seeds, and variance across runs; no standard deviations or error bars are reported for any of the accuracy or memory numbers.","section":"Implementation Details"},{"comment":"There are several typos, including 'consistancy' in the Datasets section, 'show casing' in Results and Analysis, and inconsistent spacing in references; a careful proofread is needed.","section":"Global"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style empirical report with useful resource measurements, but the evaluation protocol is currently too loose to support the headline improvement claim. The main fixes—documenting prompts/decoding for both base and finetuned models, tightening the metric, and clarifying the 48% figure—are within scope for a revision. I would not reject, but I would not accept without these clarifications. There is also a somewhat heavy reliance on self-citations to earlier FinGPT work; that is not itself a problem, but the novelty relative to Liu et al. 2024a should be stated more crisply."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: this is a workshop-grade application paper, not a methods contribution. The authors apply QLoRA to Llama 3.1 8B/70B on a set of financial tasks and report large accuracy gains over the base models plus concrete memory and speed numbers. The most useful new artifact is the XBRL extraction benchmark built from Dow Jones 30 filings; the 4-bit/rank-4 vs 8-bit/rank-8 sweep is a practical data point.\n\nWhat they do well: they position the work honestly as a follow-up to FinGPT and QLoRA, the resource tables are reproducible in principle, and the finding that 4-bit/rank-4 comes close to 8-bit/rank-8 while using less memory is a genuinely useful empirical observation. The numbers in Table 3 are internally consistent: the finetuned 8B outperforms the 70B base on most tasks, which is the kind of result that makes local finetuning attractive.\n\nWhere it's soft: the evaluation protocol is under-specified. There is no documentation of the prompts given to the base models on the classification tasks, so the stress-test concern is fair: part of the gain could be format compliance (the model learning to emit exactly 'positive' or 'negative' instead of a sentence). The scoring rule is lenient—inclusion of the ground truth anywhere in the response—and the FPB split is manually created. Only the base model is used as baseline; no comparison to full finetuning, other PEFT methods, or even regular Adam vs the 0/1 Adam optimizer. The headline 'up to 48% average increase' is not backed by the table: the average absolute gain is about 35 points, and 'up to 48%' seems to be the best single-task NER case, not an average. That should be fixed before publication.\n\nNone of these are load-bearing flaws if the paper is understood as a demo. The central claim—QLoRA improves financial LLM accuracy while cutting memory—is plausible and consistent with prior work. It just isn't established to the standard the abstract implies.\n\nMy recommendation: send it to peer review for a workshop or systems/demo track. Require the authors to document the exact prompts and decoding settings for both base and finetuned models, and to report per-task prompt templates. I'd take it as a serious referee but I'd ask for those clarifications.","headline":"Solid workshop application paper with useful memory numbers and a new XBRL extraction set, but the loose evaluation protocol keeps the 48% accuracy claim from being fully supported.","tokens_in":9104,"tokens_out":3616,"would_cite":false,"duration_ms":31147,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Finetuning Llama 3.1 models with quantized low-rank adaptation lifts accuracy on financial tasks while cutting GPU memory enough to run on a single 48GB GPU.","keywords":["QLoRA","low-rank adaptation","financial large language models","quantization","GPU memory reduction","sentiment analysis","named entity recognition","XBRL extraction"],"falsifier":"Evaluate the base Llama 3.1 8B Instruct on FPB and NER using exactly the instruction prompts, one-shot examples, and inclusion-based scoring given to the finetuned models; if its 68.73% and 48.89% scores jump toward the finetuned 86.30% and 96.63%, the claimed accuracy gains are mostly prompt-format compliance rather than learned financial skill.","tokens_in":7968,"feed_emoji":"💹","tokens_out":11563,"duration_ms":90319,"temperature":0.7,"pith_summary":"FinLoRA claims that applying QLoRA—quantizing the frozen base model to 4 or 8 bits and training only low-rank adapters—lets Llama 3.1 8B and 70B models be finetuned locally on financial tasks while using a fraction of the GPU memory. On the paper's nine financial datasets, the finetuned 8B model raises accuracy from 68.73% to 86.30% on financial sentiment, from 48.89% to 96.63% on named entity recognition, and from 2.85% to 70.45% on XBRL tagging, with an average accuracy increase of up to 48% across tasks. A sympathetic reader would care because privacy and regulatory constraints make local finetuning desirable, and the memory numbers suggest it can fit on commodity hardware: 4-bit finetuning of the 8B model uses about 8.7 GB instead of 30.9 GB, and the 70B model drops from over 300 GB to about 42.8 GB.","feed_headline":"FinLoRA: small adapters lift financial LLM accuracy up to 48 percent","feed_subtitle":"Quantized low-rank finetuning cuts GPU memory to 8.7 GB and runs financial tasks on commodity hardware.","key_machinery":"The load-bearing object is the low-rank update $\\Delta W = BA$ added to frozen pretrained weights $W_0$, making the forward pass $y = W_0 x + BA x$ with rank $r \\ll n$; QLoRA keeps $W_0$ in 4-bit or 8-bit quantized form and leaves only the small adapter matrices $A$ and $B$ in 16-bit precision. The low rank shrinks trainable parameters to about 0.039% of the original weight matrix at $n=4096$, $r=8$, and quantization shrinks resident model size; the paper adds distributed data parallelism, pipeline parallelism, BF16, and a communication-efficient optimizer to make local finetuning and inference faster and cheaper.","core_discovery":"The central discovery is empirical: quantized low-rank finetuning turns an instruction-tuned general model into a strong financial model across sentiment analysis, named entity recognition, headline classification, and XBRL tagging and extraction. With a rank-4 LoRA adapter on a 4-bit quantized Llama 3.1 8B, accuracy rises from 68.73% to 86.30% on FPB, from 48.89% to 96.63% on NER, from 45.34% to 88.03% on Headline, and from 2.85% to 70.45% on FiNER, while the 70B quantized-finetuned model reaches 80.94%, 98.88%, and 96.38% on FPB, NER, and Headline. The same configuration cuts GPU memory for 8B finetuning from 30.9 GB at 16-bit to 8.7 GB at 4-bit and for 70B from over 300 GB to 42.8 GB, and the paper states all finetuning can run on a single 48GB GPU with longer training time.","pith_inferences":["Inference: Because scoring is inclusion-based, a strict exact-match rescoring of the same outputs would separate real gains in factual precision from gains in producing verbose text that happens to contain the answer.","Inference: The paper leaves the 70B QLoRA model unevaluated on XBRL tasks; running it on FiNER, tag, and value extraction would show whether the 8B XBRL gains transfer to the larger model.","Inference: The FPB train/test split is manually created; repeating the sentiment experiments on the standard public split would indicate how much of the 68.73% to 86.30% jump depends on split choice.","Inference: Keeping adapter weights separate at inference invites a multi-adapter router that lets one quantized base model switch among sentiment, NER, headline, and XBRL tasks, which the paper names as its future mixture-of-experts direction."],"forward_implications":["A 4-bit rank-4 adapter is enough for the 8B model to reach 86.30% on FPB and 96.63% on NER, so memory-saving settings do not cost accuracy.","The QLoRA-finetuned 8B model outscores the unadapted 70B base model on all nine reported tasks, showing task adaptation can beat raw scale for financial workloads.","Finetuning the 70B model in 4-bit fits in 42.8 GB of GPU memory, which puts large-model financial finetuning within reach of a single high-end GPU.","XBRL tagging accuracy jumps from 2.85% to 70.45% and XBRL tag/value extraction from 79.37%/55.26% to 95.00%/96.05%, making local LLM-based XBRL report analysis practical."],"supporting_citations":[{"why":"Defines QLoRA, the 4-bit and 8-bit quantization with low-rank adapters that the paper's finetuning uses.","marker":"Dettmers et al. 2023"},{"why":"Defines LoRA's low-rank weight update and the parameter counts that justify the memory savings.","marker":"Hu et al. 2021"},{"why":"Provides the Llama 3.1 8B and 70B Instruct models that are finetuned and evaluated.","marker":"Dubey et al. 2024"},{"why":"Provides the Financial Phrasebank dataset used for sentiment analysis.","marker":"Malo et al. 2013"},{"why":"Provides the FiQA SA dataset for financial sentiment analysis.","marker":"Maia et al. 2018"},{"why":"Provides the NWGI sentiment dataset and the FinGPT baseline context the work builds on.","marker":"Liu et al. 2023"},{"why":"Provides the NER dataset used for entity recognition evaluation.","marker":"Salinas Alvarado, Verspoor, and Baldwin 2015"},{"why":"Provides the news Headline dataset for headline classification.","marker":"Sinha and Khandait 2020"},{"why":"Provides the FiNER dataset for XBRL tagging.","marker":"Loukas et al. 2022"},{"why":"Establishes the XBRL agent task setup for tag and value extraction that the paper evaluates against.","marker":"Han et al. 2024"}],"fun_headline_variants":["FinLoRA: 4-bit finetuning lifts financial LLM accuracy by up to 48 points","FinLoRA: single-GPU finetuning for financial LLMs with 8.7GB memory","4-bit QLoRA: financial LLM accuracy up 48 points on NER","FinLoRA cuts memory 3.5x and boosts financial accuracy to 96% NER","QLoRA enables local financial LLM finetuning on commodity GPUs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the base models received the same task instructions, one-shot examples, and lenient substring-inclusion scoring as the finetuned models, since otherwise the reported accuracy gaps could reflect output-format compliance rather than financial knowledge learned by finetuning.","fun_headline_variants_meta":{"raw":{"variants":["FinLoRA: 4-bit finetuning lifts financial LLM accuracy by up to 48 points","FinLoRA: single-GPU finetuning for financial LLMs with 8.7GB memory","4-bit QLoRA: financial LLM accuracy up 48 points on NER","FinLoRA cuts memory 3.5x and boosts financial accuracy to 96% NER","QLoRA enables local financial LLM finetuning on commodity GPUs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000888,"raw_usage":{"total_tokens":3833,"prompt_tokens":946,"completion_tokens":2887,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2767}},"tokens_in":562,"tokens_out":2887,"duration_ms":18510,"temperature":1.0,"reasoning_tokens":2767,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:59:38.807881+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the base Llama 3.1 8B Instruct on FPB and NER using exactly the instruction prompts, one-shot examples, and inclusion-based scoring given to the finetuned models; if its 68.73% and 48.89% scores jump toward the finetuned 86.30% and 96.63%, the claimed accuracy gains are mostly prompt-format compliance rather than learned financial skill.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FiQA SA dataset for financial sentiment analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NWGI sentiment dataset and the FinGPT baseline context the work builds on."},{"cited_title":"C.; Verspoor, K.; and Baldwin, T","cited_arxiv_id":null,"evidence_quote":"Provides the NER dataset used for entity recognition evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FiNER dataset for XBRL tagging."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the XBRL agent task setup for tag and value extraction that the paper evaluates against."}],"review_version":1}