{"id":"19de3532-d61c-49f6-b40a-4b36df48509f","arxiv_id":"2608.04488","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On consumer GPUs, LoRA+ gives the best energy-focused fine-tuning score in 19 of 24 small-model task configurations, while QLoRA wins the memory-focused score when peak VRAM is the binding constraint.","lead":"This paper benchmarks five fine-tuning methods on four small language models across six tasks, measuring accuracy alongside training time, memory, and energy on a consumer GPU. It finds LoRA+ usually wins when energy is the priority, QLoRA when memory is the priority, and TinyLlama-1.1B is the most efficient overall.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"QLoRA memory advantage may be underestimated because peak VRAM is only sampled at 100 ms; the 3.9x claim and 5/12 NS-M wins rest on a single run with no repetition.","rationale":"The reader's weakest_assumption targets the NetScore metric exponents and the selection rule. That is a legitimate interpretive caveat but not the most load-bearing issue: LoRA+ is selected 18/24 under an energy-first rule, and even if the metric weights changed, LoRA+ still wins raw accuracy in most configurations at identical FLOPs, so the qualitative energy conclusion is fairly robust to the S=20, alpha=2, 1/8 choices. The truly load-bearing weakness is the empirical measurement basis: (1) peak VRAM is sampled at only 100 ms via pynvml, so the very memory claims (3.9x, 5/12 NS-M wins, QLoRA as the memory choice) rest on an undersampled single measurement; and (2) every run uses one seed, so the exact counts 19/24, 18/24, 13/24 are point estimates with no variance information. These are concrete, falsifiable, and directly support the paper's core practical conclusion ('LoRA+ for energy, QLoRA for memory'). The reader noted the lack of repeated runs but tied it only to the NetScore comparison, not to the memory measurement sampling rate. I therefore recommend keeping CONDITIONAL but adding the VRAM sampling check as a required revision.","tokens_in":34488,"tokens_out":1923,"duration_ms":18466,"concrete_test":"Re-measure peak VRAM for TinyLlama-1.1B on SST-2 under QLoRA and LoRA using a 1 ms sampling loop (or torch.cuda.max_memory_allocated) over the full training run, and repeat the LoRA vs LoRA+ vs QLoRA energy runs 5 times with different seeds. If the re-measured QLoRA peak VRAM is not below the LoRA peak by a large margin, or if any of the 18/24 selections flip under run-to-run variance (NS-E differences below ~0.3), the headline count claims would need revision.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central practical conclusion is that LoRA+ wins for energy (19/24 NS-E, 18/24 selected) and QLoRA wins for memory (5/12 Transformer NS-M, up to 3.9x VRAM cut). The first half is robust to metric weighting since LoRA+ wins even on raw accuracy in many cases and costs the same FLOPs as LoRA. The load-bearing weakness is the memory half: the claimed up-to-3.9x VRAM reduction and QLoRA's 5/12 NS-M wins depend on a peak-VRAM measurement that is sampled at 100 ms via pynvml (Section IV-A). Peak VRAM sampled at 10 Hz will miss short-lived allocation spikes (optimizer states, temporary activation buffers, CUDA caching allocator growth), so the reported 'peak' values (e.g., 4,225 MB vs 16,480 MB for TinyLlama on SST-2) are lower bounds on the true peak. Additionally, each configuration was run once (fixed seed 42), so the small NS-E margins that decide selection in LoRA vs LoRA+ cases (e.g., 70.0 vs 69.9 on TinyLlama SST-2; 60.2 vs 60.1 on TinyLlama LaMP-3; QLoRA tie with LoRA+ at 63.9 on TinyLlama LaMP-2) have no error bars; if run-to-run energy or time varies by more than ~0.1-0.3%, several of the 18/24 selections and the 19/24 count could shift.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical comparison of five fine-tuning approaches (Full-FT, LoRA, LoRA+, QLoRA, BitFit) on four sub-2B SLMs (TinyLlama-1.1B, Qwen3-1.7B, Mamba-1.4B, Mamba2-1.3B) across three GLUE and three LaMP tasks. It measures accuracy, TFLOPs, time, peak VRAM, power, and energy for 108 finetuning runs on a single RTX 4090, and aggregates them through four NetScore variants (NS, NS-E, NS-M, NS#). Using an energy-first selection rule, the paper finds LoRA+ selected in 18/24 model-task configurations, QLoRA best on NS-M in 5/12 Transformer configurations, and TinyLlama-1.1B the most efficient model; it concludes the optimal method depends on the dominant constraint: LoRA+ for energy, QLoRA for memory.","tokens_in":34851,"tokens_out":6280,"duration_ms":69073,"significance":"The paper's main value is a broad, well-documented measurement matrix that includes energy and memory, two axes usually missing from PEFT comparisons; the cross-family Transformer vs SSM comparison and the public code release are strengths. If the conclusions survive the robustness concerns below, the practical recommendation (LoRA+ when energy is the constraint, QLoRA when VRAM is binding, TinyLlama as a default base model) would be directly actionable for on-device deployment.","major_comments":[{"comment":"Peak VRAM is monitored with pynvml sampled every 100 ms, so short-lived allocation spikes (optimizer states, temporary activation buffers, CUDA caching allocator growth) can be missed. The reported peaks (e.g., 4,225 MB vs 16,480 MB for TinyLlama on SST-2 in Table IV) are therefore lower bounds, and the 3.9x VRAM reduction claim as well as QLoRA's 5/12 NS-M wins rest on a measurement that may overstate QLoRA's advantage. Please supplement or replace with CUDA-level peak tracking (torch.cuda.max_memory_allocated) over the whole run and, if possible, report high-frequency samples around optimizer steps.","section":"Section IV-A, Table II"},{"comment":"Each configuration is run once with a fixed seed, and training time, energy, and peak VRAM all vary across runs on the same GPU. Many selections are decided by NS-E gaps of 0.1-0.3 points (TinyLlama SST-2 LoRA 70.0 vs LoRA+ 69.9; TinyLlama LaMP-3 LoRA 60.2 vs LoRA+ 60.1; TinyLlama LaMP-2 QLoRA tie with LoRA+ at 63.9). Without repeated runs or error bars, the headline counts of 19 of 24 for NS-E and 18 of 24 for selection are not established as stable. At minimum, repeat the close configurations several times and report means and ranges.","section":"Section V-A, Tables IV and V"},{"comment":"The NetScore formula and its exponents (S=20, alpha=2, 1/8 efficiency exponents) are adopted from prior work without sensitivity analysis. Because the selection rule is defined solely on NS-E, the dominance of LoRA+ under the energy-first rule is a statement about one particular weighting. The raw data in Tables IV and V would allow a sensitivity check; please add an analysis varying alpha and the efficiency exponent (e.g., 1/4) to show that the 18/24 selection count and the LoRA+ vs LoRA near-ties are robust.","section":"Section III-A, Table I"}],"minor_comments":[{"comment":"The inference column headers and the inline unit notes are inconsistent: the caption says LaMP-1/LaMP-3 inference time is in minutes and LaMP-2 in seconds, while the column headers only say Time and Energy. Please put the units directly in the headers or use a consistent legend.","section":"Table V caption"},{"comment":"The word 'substaintially' should be 'substantially'.","section":"Section V-B2"},{"comment":"The capitalization of LoRA is inconsistent ('LoRa' appears in several places); please standardize to 'LoRA'.","section":"Section II-C"},{"comment":"The NetScore formula is not numbered; numbering it would make the references to the formula in the text and tables easier to follow.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The measurement methodology issues are not fatal; they require additional data collection. The paper's self-stated limitations (single GPU, no QLoRA for SSMs) are honestly acknowledged. I would ask the authors to provide torch.cuda.max_memory_allocated values or equivalent, repeated runs for close configurations, and a NetScore weighting sensitivity analysis; after that, the paper would likely be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a careful, honest empirical benchmark of five PEFT methods on four small models across two architecture families, with energy, memory, time, and accuracy all measured. The broad conclusions—LoRA+ for energy, QLoRA for memory, BitFit only when you care about the parameter-count-based NetScore—are believable and practically useful. But the precise headline counts (19/24, 18/24, 5/12) are built on single runs and hair-thin margins, so don't quote them as law.\n\nWhat's actually new: to my knowledge, this is the first comparison that puts Transformer and SSM families through the same PEFT pipeline on both GLUE and LaMP tasks while recording power and energy. The tables are thorough, the code is released, and the selection rule (energy-first, NS# tie-break) is transparent. The LoRA+ result is mechanically plausible: same rank and FLOPs as LoRA, better convergence, so any accuracy gain costs nothing. QLoRA's memory win is consistent with prior work. The authors also honestly flag the single-GPU setup and the missing QLoRA support for Mamba.\n\nSoft spots. The lack of repeated runs is the real one. Many selections are decided by differences under half a percent (70.0 vs 69.9; 63.9 tie). With a single seed, those counts are fragile. The peak VRAM measurement samples at 100 ms, which will undercatch transient spikes; the QLoRA-vs-LoRA gap is large enough that the qualitative memory conclusion survives, but the exact 3.9x and 5/12 count are not exact. The NetScore exponents are taken from prior work without sensitivity analysis, so the metric weighting is a choice, not a finding. The Mamba-vs-Transformer comparisons are contaminated by software maturity, which the paper acknowledges. None of this is fatal; the direction of every conclusion is probably right.\n\nWho it's for: practitioners picking a PEFT method for a single-GPU on-device scenario. It's a useful reference table, not a new idea.\n\nRecommendation: accept for peer review. A good referee should ask for error bars or at least multiple seeds, a more careful VRAM measurement protocol, and some robustness checks on the NetScore exponents. My own verdict would be conditional, but it's a legitimate contribution and deserves referee time.","headline":"Useful, honest PEFT benchmark on small models; the broad energy/memory conclusions hold, but the exact win counts rest on single runs and tiny margins.","tokens_in":35321,"tokens_out":3157,"would_cite":true,"duration_ms":34943,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Comparing five fine-tuning methods on four 1-2B models across six benchmarks, this paper finds that LoRA+ wins the energy-focused score in 19 of 24 configurations and is the energy-first selection in 18 of 24, with QLoRA preferred only…","keywords":["Parameter-Efficient Fine-Tuning","Small Language Models","LoRA+","QLoRA","NetScore-E","NetScore-M","on-device personalization","energy-aware model selection"],"falsifier":"Recompute all 24 NetScore-E rankings with the efficiency exponent set to 1/4 instead of 1/8 (or with alpha=1 instead of 2); if LoRA+ no longer wins the large majority of configurations, the selection rule's headline result is an artifact of the metric settings. A simpler check is to run the same 24 configurations on a second consumer GPU with different power characteristics and see whether the energy rankings persist.","tokens_in":34324,"feed_emoji":"⚡","tokens_out":7216,"duration_ms":67865,"temperature":0.7,"pith_summary":"This paper asks which parameter-efficient fine-tuning method a developer should use to adapt a 1-2 billion parameter language model on a single consumer GPU, when energy and memory are the constraints that matter. The authors run five fine-tuning approaches (full fine-tuning, LoRA, LoRA+, QLoRA, and BitFit) on four small models from two architecture families across three general-understanding and three personalization tasks, measuring accuracy, time, VRAM, and energy. They find that LoRA+ wins the energy-focused composite score in 19 of 24 model-task configurations, making it the energy-first selection in 18 of 24, while QLoRA wins the memory-focused score in 5 of 12 Transformer configurations. TinyLlama-1.1B is the most efficient model overall. The paper's practical conclusion is that compact models combined with PEFT are a viable energy-aware path to on-device personalization, with the best method set by the dominant constraint.","feed_headline":"LoRA+ wins 18 of 24 energy-first picks for on-device SLM tuning","feed_subtitle":"On one consumer GPU, the best PEFT method depends on the binding constraint: LoRA+ for energy, QLoRA for VRAM.","key_machinery":"The load-bearing object is the NetScore family of composite metrics, defined as $\\text{NetScore} = S \\log_{10}\\left( a^{\\alpha} / \\left( (p m)^{\\beta} v^{\\gamma} t^{\\delta} w^{\\lambda} \\right) \\right)$, with $S=20$ and $\\alpha=2$ and efficiency exponents set to $1/8$. Toggling the exponents $\\beta, \\gamma, \\delta, \\lambda$ yields the variants used for selection: NetScore-E (time and power, $\\delta=\\lambda=1/8$), NetScore-M (peak VRAM, $\\gamma=1/8$), and NetScore# (VRAM, time, and power all at $1/8$). The energy-first selection rule—highest NetScore-E, ties broken by NetScore#—is what turns LoRA+'s essentially free accuracy gain into a win in 18 of 24 configurations. The second mechanism is LoRA+'s differentiated learning rates for the $A$ and $B$ low-rank matrices ($\\text{lr}_B = 16 \\cdot \\text{lr}_A$), which accelerates convergence under short training budgets without changing parameter count or FLOPs.","core_discovery":"The paper's central claim is that no single fine-tuning method dominates; the right choice depends on the binding hardware constraint, and the paper provides a head-to-head measurement to make that choice. Under a strict energy-first selection rule (highest NetScore-E, ties broken by NetScore#), LoRA+ is selected in 18 of 24 model-task pairs and achieves the highest NetScore-E in 19 of 24. QLoRA, available only for Transformer models, cuts peak fine-tuning VRAM by up to 3.9x relative to LoRA and holds the highest memory-focused NetScore-M in 5 of 12 Transformer configurations, but its de-quantization overhead makes it energy-selected only once. Full fine-tuning is selected exactly once, and BitFit is never selected on either efficiency variant. The authors interpret these results as evidence that PEFT on SLMs matches or beats full fine-tuning at a fraction of the resource cost, and that TinyLlama-1.1B is the most efficient model on most benchmarks.","pith_inferences":["If the NetScore exponent choices were varied (for example, efficiency weight 1/4 instead of 1/8, or a different performance exponent), the LoRA+ dominance could shrink; the paper does not test this sensitivity, so the headline ranking is conditional on that metric choice.","The reported inference measurements imply that deployment decisions should include inference energy, not just fine-tuning: QLoRA adds dequantization latency and energy, and generative LaMP inference can take much longer than classification inference.","The SSM software gap is plausibly a moving target; as Mamba kernels mature, the energy rankings between Transformer and SSM families could shift, so the 'TinyLlama always wins' result may be specific to the current library stack.","A natural extension is to test whether the same constraint-based rule (energy-first, memory-first) transfers to smaller mobile GPUs or NPUs with 4-8GB memory, where QLoRA's VRAM advantage could become decisive for all models."],"forward_implications":["Developers with an energy budget should default to LoRA+ for 1-2B SLMs on a single consumer GPU; it matched or beat full fine-tuning in accuracy in most configurations while using far less energy.","When peak VRAM is the hard limit, QLoRA is the answer for Transformer models: up to 3.9x lower fine-tuning VRAM than LoRA, at the price of longer training and higher energy.","Full fine-tuning and BitFit are not competitive under either efficiency lens: full fine-tuning is selected once in 24 pairs, BitFit never, and BitFit collapses on regression and personalization tasks.","SSMs (Mamba/Mamba2) gain more from LoRA+ than Transformers do, but their longer training times and less optimized kernels erase that advantage in NetScore terms on most tasks.","The choice of the NetScore variant changes the optimal model: TinyLlama leads on base and memory variants, while Qwen3 can win on energy-focused LaMP-1 when accuracy outweighs energy cost."],"supporting_citations":[{"why":"Supplies the base NetScore formula and the S=20, alpha=2 settings used by every variant.","marker":"[17]"},{"why":"Defines the energy-focused NetScore-E variant that drives the paper's selection rule.","marker":"[19]"},{"why":"Defines the memory-focused NetScore-M and combined NetScore# variants, including the tie-break metric.","marker":"[21]"},{"why":"Supplies the LoRA adapter configuration (rank 16, alpha 32) applied to attention and SSM projections.","marker":"[22]"},{"why":"Supplies LoRA+ with its differential A/B learning rates, the method selected in 18 of 24 configurations.","marker":"[23]"},{"why":"Supplies QLoRA 4-bit NF4 quantization, the method that sets the memory-focused records and the de-quantization overhead measured.","marker":"[24]"},{"why":"Supplies BitFit, the bias-only baseline that the paper finds non-competitive.","marker":"[25]"},{"why":"Supplies TinyLlama-1.1B, the model that leads the NetScore rankings on most benchmarks.","marker":"[27]"},{"why":"Supplies the Mamba SSM architecture whose PEFT behavior is compared against Transformers.","marker":"[31]"},{"why":"Supplies the LaMP personalization tasks and their generative fine-tuning protocol.","marker":"[34]"}],"fun_headline_variants":["LoRA+ wins 18/24 energy picks on SLMs","QLoRA cuts VRAM 3.9x, LoRA+ wins energy","PEFT tradeoff: LoRA+ energy, QLoRA memory","LoRA+ dominates energy-first SLM tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire ranking depends on the particular way the paper's score weighs accuracy against energy and memory; if a different but equally reasonable weighting were used, the winning method could change.","fun_headline_variants_meta":{"raw":{"variants":["LoRA+ wins 18/24 energy picks on SLMs","QLoRA cuts VRAM 3.9x, LoRA+ wins energy","PEFT tradeoff: LoRA+ energy, QLoRA memory","LoRA+ dominates energy-first SLM tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1395,"prompt_tokens":1199,"completion_tokens":196,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":815,"completion_tokens_details":{"reasoning_tokens":120}},"tokens_in":815,"tokens_out":196,"duration_ms":2701,"temperature":1.0,"reasoning_tokens":120,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:20:58.653300+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute all 24 NetScore-E rankings with the efficiency exponent set to 1/4 instead of 1/8 (or with alpha=1 instead of 2); if LoRA+ no longer wins the large majority of configurations, the selection rule's headline result is an artifact of the metric settings. A simpler check is to run the same 24 configurations on a second consumer GPU with different power characteristics and see whether the energy rankings persist.","supporting_citations":[{"cited_title":"How green is continual learning, really? Analyzing the energy consumption in continual training of vision foundation models","cited_arxiv_id":"2409.18664","evidence_quote":"Defines the energy-focused NetScore-E variant that drives the paper's selection rule."},{"cited_title":"Efficient PEFT Methods with Adaptive Checkpointing for Vision Models and VLMs on Resource Constrained Consumer-GPUs","cited_arxiv_id":"2607.02158","evidence_quote":"Defines the memory-focused NetScore-M and combined NetScore# variants, including the tie-break metric."},{"cited_title":"Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,","cited_arxiv_id":null,"evidence_quote":"Supplies BitFit, the bias-only baseline that the paper finds non-competitive."}],"review_version":1}