Pith. sign in

REVIEW 3 major objections 4 minor 46 references

Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Comparing five fine-tuning methods on four 1-2B models across six benchmarks, this paper finds that LoRA+ wins the energy-focused score in 19 of 24 configurations and is the energy-first selection in 18 of 24, with QLoRA preferred only…

desk verdict Useful, honest PEFT benchmark on small models; the broad energy/memory conclusions hold, but the exact win counts rest on single runs and tiny margins. read the letter →

arxiv 2608.04488 v1 pith:JJOWR5I4 submitted 2026-08-05 cs.CL

Kuanysh Akhmetzhanov , Jurn-Gyu Park This is my paper
classification cs.CL
keywords Parameter-EfficientFine-TuningSmallLanguageModelsLoRA+QLoRANetScore-EScore-Mon-devicepersonalizationenergy-awaremodelselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks which parameter-efficient fine-tuning method a developer should use to adapt a 1-2 billion parameter language model on a single consumer GPU, when energy and memory are the constraints that matter. The authors run five fine-tuning approaches (full fine-tuning, LoRA, LoRA+, QLoRA, and BitFit) on four small models from two architecture families across three general-understanding and three personalization tasks, measuring accuracy, time, VRAM, and energy. They find that LoRA+ wins the energy-focused composite score in 19 of 24 model-task configurations, making it the energy-first selection in 18 of 24, while QLoRA wins the memory-focused score in 5 of 12 Transformer configurations. TinyLlama-1.1B is the most efficient model overall. The paper's practical conclusion is that compact models combined with PEFT are a viable energy-aware path to on-device personalization, with the best method set by the dominant constraint.

What carries the argument

The load-bearing object is the NetScore family of composite metrics, defined as $\text{NetScore} = S \log_{10}\left( a^{\alpha} / \left( (p m)^{\beta} v^{\gamma} t^{\delta} w^{\lambda} \right) \right)$, with $S=20$ and $\alpha=2$ and efficiency exponents set to $1/8$. Toggling the exponents $\beta, \gamma, \delta, \lambda$ yields the variants used for selection: NetScore-E (time and power, $\delta=\lambda=1/8$), NetScore-M (peak VRAM, $\gamma=1/8$), and NetScore# (VRAM, time, and power all at $1/8$). The energy-first selection rule—highest NetScore-E, ties broken by NetScore#—is what turns LoRA+'s essentially free accuracy gain into a win in 18 of 24 configurations. The second mechanism is LoRA+'s differentiated learning rates for the $A$ and $B$ low-rank matrices ($\text{lr}_B = 16 \cdot \text{lr}_A$), which accelerates convergence under short training budgets without changing parameter count or FLOPs.

What would settle it

Recompute all 24 NetScore-E rankings with the efficiency exponent set to 1/4 instead of 1/8 (or with alpha=1 instead of 2); if LoRA+ no longer wins the large majority of configurations, the selection rule's headline result is an artifact of the metric settings. A simpler check is to run the same 24 configurations on a second consumer GPU with different power characteristics and see whether the energy rankings persist.

Watch

Extended reading notes

Core claim

The paper's central claim is that no single fine-tuning method dominates; the right choice depends on the binding hardware constraint, and the paper provides a head-to-head measurement to make that choice. Under a strict energy-first selection rule (highest NetScore-E, ties broken by NetScore#), LoRA+ is selected in 18 of 24 model-task pairs and achieves the highest NetScore-E in 19 of 24. QLoRA, available only for Transformer models, cuts peak fine-tuning VRAM by up to 3.9x relative to LoRA and holds the highest memory-focused NetScore-M in 5 of 12 Transformer configurations, but its de-quantization overhead makes it energy-selected only once. Full fine-tuning is selected exactly once, and BitFit is never selected on either efficiency variant. The authors interpret these results as evidence that PEFT on SLMs matches or beats full fine-tuning at a fraction of the resource cost, and that TinyLlama-1.1B is the most efficient model on most benchmarks.

Load-bearing premise

The entire ranking depends on the particular way the paper's score weighs accuracy against energy and memory; if a different but equally reasonable weighting were used, the winning method could change.

Editorial extensions

If this is right

  • Developers with an energy budget should default to LoRA+ for 1-2B SLMs on a single consumer GPU; it matched or beat full fine-tuning in accuracy in most configurations while using far less energy.
  • When peak VRAM is the hard limit, QLoRA is the answer for Transformer models: up to 3.9x lower fine-tuning VRAM than LoRA, at the price of longer training and higher energy.
  • Full fine-tuning and BitFit are not competitive under either efficiency lens: full fine-tuning is selected once in 24 pairs, BitFit never, and BitFit collapses on regression and personalization tasks.
  • SSMs (Mamba/Mamba2) gain more from LoRA+ than Transformers do, but their longer training times and less optimized kernels erase that advantage in NetScore terms on most tasks.
  • The choice of the NetScore variant changes the optimal model: TinyLlama leads on base and memory variants, while Qwen3 can win on energy-focused LaMP-1 when accuracy outweighs energy cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the NetScore exponent choices were varied (for example, efficiency weight 1/4 instead of 1/8, or a different performance exponent), the LoRA+ dominance could shrink; the paper does not test this sensitivity, so the headline ranking is conditional on that metric choice.
  • The reported inference measurements imply that deployment decisions should include inference energy, not just fine-tuning: QLoRA adds dequantization latency and energy, and generative LaMP inference can take much longer than classification inference.
  • The SSM software gap is plausibly a moving target; as Mamba kernels mature, the energy rankings between Transformer and SSM families could shift, so the 'TinyLlama always wins' result may be specific to the current library stack.
  • A natural extension is to test whether the same constraint-based rule (energy-first, memory-first) transfers to smaller mobile GPUs or NPUs with 4-8GB memory, where QLoRA's VRAM advantage could become decisive for all models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper presents an empirical comparison of five fine-tuning approaches (Full-FT, LoRA, LoRA+, QLoRA, BitFit) on four sub-2B SLMs (TinyLlama-1.1B, Qwen3-1.7B, Mamba-1.4B, Mamba2-1.3B) across three GLUE and three LaMP tasks. It measures accuracy, TFLOPs, time, peak VRAM, power, and energy for 108 finetuning runs on a single RTX 4090, and aggregates them through four NetScore variants (NS, NS-E, NS-M, NS#). Using an energy-first selection rule, the paper finds LoRA+ selected in 18/24 model-task configurations, QLoRA best on NS-M in 5/12 Transformer configurations, and TinyLlama-1.1B the most efficient model; it concludes the optimal method depends on the dominant constraint: LoRA+ for energy, QLoRA for memory.

Significance. The paper's main value is a broad, well-documented measurement matrix that includes energy and memory, two axes usually missing from PEFT comparisons; the cross-family Transformer vs SSM comparison and the public code release are strengths. If the conclusions survive the robustness concerns below, the practical recommendation (LoRA+ when energy is the constraint, QLoRA when VRAM is binding, TinyLlama as a default base model) would be directly actionable for on-device deployment.

major comments (3)
  1. [Section IV-A, Table II] Peak VRAM is monitored with pynvml sampled every 100 ms, so short-lived allocation spikes (optimizer states, temporary activation buffers, CUDA caching allocator growth) can be missed. The reported peaks (e.g., 4,225 MB vs 16,480 MB for TinyLlama on SST-2 in Table IV) are therefore lower bounds, and the 3.9x VRAM reduction claim as well as QLoRA's 5/12 NS-M wins rest on a measurement that may overstate QLoRA's advantage. Please supplement or replace with CUDA-level peak tracking (torch.cuda.max_memory_allocated) over the whole run and, if possible, report high-frequency samples around optimizer steps.
  2. [Section V-A, Tables IV and V] Each configuration is run once with a fixed seed, and training time, energy, and peak VRAM all vary across runs on the same GPU. Many selections are decided by NS-E gaps of 0.1-0.3 points (TinyLlama SST-2 LoRA 70.0 vs LoRA+ 69.9; TinyLlama LaMP-3 LoRA 60.2 vs LoRA+ 60.1; TinyLlama LaMP-2 QLoRA tie with LoRA+ at 63.9). Without repeated runs or error bars, the headline counts of 19 of 24 for NS-E and 18 of 24 for selection are not established as stable. At minimum, repeat the close configurations several times and report means and ranges.
  3. [Section III-A, Table I] The NetScore formula and its exponents (S=20, alpha=2, 1/8 efficiency exponents) are adopted from prior work without sensitivity analysis. Because the selection rule is defined solely on NS-E, the dominance of LoRA+ under the energy-first rule is a statement about one particular weighting. The raw data in Tables IV and V would allow a sensitivity check; please add an analysis varying alpha and the efficiency exponent (e.g., 1/4) to show that the 18/24 selection count and the LoRA+ vs LoRA near-ties are robust.
minor comments (4)
  1. [Table V caption] The inference column headers and the inline unit notes are inconsistent: the caption says LaMP-1/LaMP-3 inference time is in minutes and LaMP-2 in seconds, while the column headers only say Time and Energy. Please put the units directly in the headers or use a consistent legend.
  2. [Section V-B2] The word 'substaintially' should be 'substantially'.
  3. [Section II-C] The capitalization of LoRA is inconsistent ('LoRa' appears in several places); please standardize to 'LoRA'.
  4. [Section III-A] The NetScore formula is not numbered; numbering it would make the references to the formula in the text and tables easier to follow.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's rankings are direct measurements scored by an explicitly defined metric, and the sole self-citation is non-load-bearing.

full rationale

The paper is an empirical benchmark rather than a derivation. The central claims (LoRA+ highest NetScore-E in 19/24 configurations, QLoRA highest NetScore-M in 5/12 Transformer cases, and the up-to-3.9x VRAM reduction) are computed from independently measured accuracy, time, VRAM, power, and energy via the explicitly stated NetScore formula in Section III (Equation 1 and Table I). No parameter is fitted to the outcome, and no 'prediction' is derived from the metric that is not a direct score of measured values. The selection rule is transparently applied in Section V-A, and the LoRA+/QLoRA conclusions follow the raw measurements rather than being forced by the metric's construction. The only self-citation, reference [21] by co-author J.-G. Park, defines NetScore-M and NetScore#, but the paper restates the complete coefficient table and formula in Section III, so the rankings do not depend on the citation as an unverified premise. The observation that BitFit wins the base NetScore is explicitly attributed to its minimal trainable-parameter count and the parameter term in the base metric, and the paper explicitly declines to use base NetScore for selection because it can favor methods with poor task performance; this is an acknowledged definitional consequence, not a hidden circular step. There is no self-definitional step, no fitted input renamed as prediction, and no author-imported uniqueness theorem. The empirical comparison is self-contained against external benchmarks, so the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its load-bearing components are the NetScore constants and the fixed training hyperparameters.

free parameters (4)
  • NetScore constants S, alpha, efficiency exponents = S=20, alpha=2, efficiency exponent 1/8
    These constants define the metric that determines the method ranking. They are taken from prior work or chosen by the authors without sensitivity analysis, so the headline rankings depend on them.
  • LoRA rank and alpha = r=16, alpha=32, dropout 0.1
    A single adapter configuration is used for all models and tasks, which may favor or disfavor particular architectures.
  • LoRA+ learning rate ratio = 16x
    The ratio is taken from the LoRA+ paper and fixed for all runs; the benefit of LoRA+ may be sensitive to this value.
  • LaMP retrieval k = k=4
    The number of retrieved profile entries per query is fixed to 4, which affects the difficulty of the personalization tasks.
assumptions (3)
  • domain assumption The NetScore formula is an appropriate summary of deployment quality.
    The conclusions are driven by this composite metric, and its weighting of accuracy versus time, power, and memory is assumed without external validation.
  • domain assumption The selected GLUE and LaMP tasks are representative of general and personalized on-device workloads.
    Only three tasks are chosen from each benchmark family, and the results may not generalize to other tasks.
  • domain assumption A single RTX 4090 is a representative consumer GPU.
    The paper explicitly limits its conclusions to this GPU, but the practical recommendations assume transfer to other consumer hardware.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs." pith.science (2026). https://pith.science/paper/JJOWR5I4

@misc{pith2026260804488,
  author       = {Pith},
  title        = {Pith review of: Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JJOWR5I4}},
  note         = {Machine review of arXiv:2608.04488}
}
read the original abstract

Despite rapid advances in large language models (LLMs), deploying and personalizing them on resource-constrained devices remains impractical due to high VRAM, time, and energy costs. Parameter-Efficient Fine-Tuning (PEFT) of Small Language Models (SLMs) offers a promising alternative, yet few studies compare PEFT methods across architectures using both general and personalization benchmarks while accounting for energy consumption. We compare five fine-tuning approaches (Full Fine-Tuning, LoRA, LoRA+, QLoRA, and BitFit) on four SLMs from two families (Transformer-based: TinyLlama-1.1B, Qwen3-1.7B; SSM-based: Mamba-1.4B, Mamba-2-1.3B) across three GLUE tasks (SST-2, QNLI, STS-B) and three LaMP personalization tasks (LaMP-1, LaMP-2, LaMP-3). Each configuration is evaluated with the energy-focused NetScore-E and the memory-focused NetScore-M, the two variants that reflect the constraints binding on-device deployment. Methods are selected with a strict energy-first rule (highest NetScore-E, ties broken by NetScore#). LoRA+ achieves the highest NetScore-E in 19 of 24 configurations and the highest NetScore-M in 13 of 24, and is the selected method in 18 of 24. QLoRA, available only for the Transformer models, cuts peak finetuning VRAM by up to 3.9x relative to LoRA and therefore takes the best NetScore-M in 5 of the 12 Transformer configurations, although its de-quantization overhead leaves it selected in only one of them once energy decides. BitFit and full fine-tuning are almost never competitive on either variant, and TinyLlama-1.1B leads the energy-focused NetScore-E on five of the six benchmarks and the memory-focused NetScore-M on four. These results show that compact SLMs paired with PEFT provide a practical, energy-aware path to personalized on-device deployment, with the optimal method set by the dominant constraint: LoRA+ for energy and QLoRA for memory.

Figures

Figures reproduced from arXiv: 2608.04488 by the authors.

Figure 1
Figure 1. Methodology Overview. Fine-Tuning Methods. Each model is adapted to every benchmark task using five fine-tuning methods: Full Fine￾Tuning (Full-FT), which updates all parameters, and four PEFT methods, which are LoRA, LoRA+, QLoRA (4-bit quantization), and BitFit. This results in 18 configurations per task (4 models × 5 methods, minus the two QLoRA runs that the available libraries do not support for the SSM models)… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 10 canonical work pages

  1. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023. [Online]. Available: https://arxiv.org/abs/1706.03762

    arXiv 2023

  2. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019. [Online]. Available: https://arxiv.org/abs/1810.04805

    arXiv 2019

  3. [3]

    Generative pre-trained transformer: A comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions,

    G. Yenduri, R. M, C. S. G, S. Y , G. Srivastava, P. K. R. Maddikunta, D. R. G, R. H. Jhaveri, P. B, W. Wang, A. V . Vasilakos, and T. R. Gadekallu, “Generative pre-trained transformer: A comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions,” 2023. [Online]. Available: https://arxiv.org/abs/2305.10435

    arXiv 2023

  4. [4]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, “Training compute-optimal large language models,” 2022. [Online]. Available: ...

    arXiv 2022

  5. [5]

    A cost-benefit analysis of on-premise large language model deployment: Breaking even with commercial llm services,

    G. Pan, V . Chodnekar, A. Roy, and H. Wang, “A cost-benefit analysis of on-premise large language model deployment: Breaking even with commercial llm services,” 2025. [Online]. Available: https://arxiv.org/abs/2509.18101

    arXiv 2025

  6. [6]

    Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Production

    C. Irugalbandara, A. Mahendra, R. Daynauth, T. K. Arachchige, J. Dantanarayana, K. Flautner, L. Tang, Y . Kang, and J. Mars, “Scaling down to scale up: A cost-benefit analysis of replacing openai’s llm with open source slms in production,” 2024. [Online]. Available: https://arxiv.org/abs/2312.14972

    work page Pith review arXiv 2024

  7. [7]

    Understanding the performance and estimating the cost of llm fine-tuning,

    Y . Xia, J. Kim, Y . Chen, H. Ye, S. Kundu, C. Hao, and N. Talati, “Understanding the performance and estimating the cost of llm fine-tuning,” 2024. [Online]. Available: https://arxiv.org/abs/2408.04693

    arXiv 2024

  8. [8]

    Small language models are the future of agentic ai,

    P. Belcak, G. Heinrich, S. Diao, Y . Fu, X. Dong, S. Muralidharan, Y . C. Lin, and P. Molchanov, “Small language models are the future of agentic ai,” 2025. [Online]. Available: https://arxiv.org/abs/2506.02153

    arXiv 2025

Show all 46 references
  1. [9]

    A survey on small language models,

    C. V . Nguyen, X. Shen, R. Aponte, Y . Xia, S. Basu, Z. Hu, J. Chen, M. Parmar, S. Kunapuli, J. Barrow, J. Wu, A. Singh, Y . Wang, J. Gu, N. K. Ahmed, N. Lipka, R. Zhang, X. Chen, T. Yu, S. Kim, H. Deilamsalehy, N. Park, M. Rimer, Z. Zhang, H. Yang, P. Mathur, G. Wu, F. Dernon...

    2025

  2. [10]

    It’s not just size that matters: Small language models are also few-shot learners,

    T. Schick and H. Sch ¨utze, “It’s not just size that matters: Small language models are also few-shot learners,” inProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, K. Toutanova, A. R...

    2021

  3. [11]

    Small language models: Survey, measurements, and insights,

    Z. Lu, X. Li, D. Cai, R. Yi, F. Liu, X. Zhang, N. D. Lane, and M. Xu, “Small language models: Survey, measurements, and insights,” 2025. [Online]. Available: https://arxiv.org/abs/2409.15790 PREPRINT, 2026 14

    2025 arXiv

  4. [12]

    Small language models (slms) can still pack a punch: A survey,

    S. Subramanian, V . Elango, and M. Gungor, “Small language models (slms) can still pack a punch: A survey,” 2025. [Online]. Available: https://arxiv.org/abs/2501.05465

    2025 arXiv

  5. [13]

    Empirical analysis of the strengths and weaknesses of peft techniques for llms,

    G. Pu, A. Jain, J. Yin, and R. Kaplan, “Empirical analysis of the strengths and weaknesses of peft techniques for llms,” 2023. [Online]. Available: https://arxiv.org/abs/2304.14999

    2023 arXiv

  6. [14]

    Peft-u: Parameter- efficient fine-tuning for user personalization,

    C. Clarke, Y . Heng, L. Tang, and J. Mars, “Peft-u: Parameter- efficient fine-tuning for user personalization,” 2024. [Online]. Available: https://arxiv.org/abs/2407.18078

    2024 arXiv

  7. [15]

    Parameter efficient fine tuning: A comprehensive analysis across applications,

    C. C. S. Balne, S. Bhaduri, T. Roy, V . Jain, and A. Chadha, “Parameter efficient fine tuning: A comprehensive analysis across applications,”

  8. [16]

    Parameter-efficient fine-tuning for foundation models,

    D. Zhang, T. Feng, L. Xue, Y . Wang, Y . Dong, and J. Tang, “Parameter-efficient fine-tuning for foundation models,” 2025. [Online]. Available: https://arxiv.org/abs/2501.13787

    2025 arXiv

  9. [17]

    Netscore: Towards universal metrics for large-scale performance analysis of deep neural networks for practical on-device edge usage,

    A. Wong, “Netscore: Towards universal metrics for large-scale performance analysis of deep neural networks for practical on-device edge usage,” 2018. [Online]. Available: https://arxiv.org/abs/1806.05512

    2018 arXiv

  10. [18]

    Attonets: Compact and efficient deep neural networks for the edge via human-machine collaborative design,

    A. Wong, Z. Q. Lin, and B. Chwyl, “Attonets: Compact and efficient deep neural networks for the edge via human-machine collaborative design,” 2019. [Online]. Available: https://arxiv.org/abs/1903.07209

    2019 arXiv

  11. [19]

    How green is continual learning, really? analyzing the energy consumption in continual training of vision foundation models,

    T. Trinci, S. Magistri, R. Verdecchia, and A. D. Bagdanov, “How green is continual learning, really? analyzing the energy consumption in continual training of vision foundation models,” 2024. [Online]. Available: https://arxiv.org/abs/2409.18664

    2024 arXiv

  12. [20]

    An attention-based feature memory design for energy-efficient continual learning,

    Y . Wang, F. Gunnarsson, and R. Hai, “An attention-based feature memory design for energy-efficient continual learning,” 2026. [Online]. Available: https://arxiv.org/abs/2510.04660

    2026

  13. [21]

    Efficient peft methods with adaptive checkpointing for vision models and vlms on resource constrained consumer-gpus,

    A. Toktassyn and J.-G. Park, “Efficient peft methods with adaptive checkpointing for vision models and vlms on resource constrained consumer-gpus,”arXiv preprint arXiv:2607.02158, 2026

    2026 arXiv

  14. [22]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,”

  15. [23]

    Lora+: Efficient low rank adaptation of large models,

    S. Hayou, N. Ghosh, and B. Yu, “Lora+: Efficient low rank adaptation of large models,” 2024. [Online]. Available: https: //arxiv.org/abs/2402.12354

    2024 arXiv

  16. [24]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” 2023. [Online]. Available: https://arxiv.org/abs/2305.14314

    2023 arXiv

  17. [25]

    Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,

    E. Ben-Zaken, S. Ravfogel, and Y . Goldberg, “Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,”

  18. [26]

    A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and trustworthiness,

    F. Wang, Z. Zhang, X. Zhang, Z. Wu, T. Mo, Q. Lu, W. Wang, R. Li, J. Xu, X. Tang, Q. He, Y . Ma, M. Huang, and S. Wang, “A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and tr...

    2024 arXiv

  19. [27]

    Tinyllama: An open-source small language model,

    P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,” 2024. [Online]. Available: https://arxiv.org/abs/2401.02385

    2024 arXiv

  20. [28]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. L...

  21. [29]

    Mobilellm: Optimizing sub-billion parameter language models for on-device use cases,

    Z. Liu, C. Zhao, F. Iandola, C. Lai, Y . Tian, I. Fedorov, Y . Xiong, E. Chang, Y . Shi, R. Krishnamoorthi, L. Lai, and V . Chandra, “Mobilellm: Optimizing sub-billion parameter language models for on-device use cases,” 2024. [Online]. Available: https: //arxiv.org/abs/2402.14905

    2024 arXiv

  22. [30]

    Demystifying small language models for edge deployment,

    Z. Lu, X. Li, D. Cai, R. Yi, F. Liu, W. Liu, J. Luan, X. Zhang, N. Lane, and M. Xu, “Demystifying small language models for edge deployment,” 01 2025, pp. 14 747–14 764

    2025

  23. [31]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” 2024. [Online]. Available: https://arxiv.org/abs/ 2312.00752

    2024 arXiv

  24. [32]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality,

    T. Dao and A. Gu, “Transformers are ssms: Generalized models and efficient algorithms through structured state space duality,” 2024. [Online]. Available: https://arxiv.org/abs/2405.21060

    2024 arXiv

  25. [33]

    Glue: A multi-task benchmark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” 2019. [Online]. Available: https://arxiv.org/abs/1804.07461

    2019 arXiv

  26. [34]

    Lamp: When large language models meet personalization,

    A. Salemi, S. Mysore, M. Bendersky, and H. Zamani, “Lamp: When large language models meet personalization,” 2024. [Online]. Available: https://arxiv.org/abs/2304.11406

    2024 arXiv

  27. [35]

    Pid parameters optimization by using genetic algorithm,

    A. Mirzal, S. Yoshii, and M. Furukawa, “Pid parameters optimization by using genetic algorithm,” 2012. [Online]. Available: https: //arxiv.org/abs/1204.0885

    2012 arXiv

  28. [36]

    Squad: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,” 2016. [Online]. Available: https://arxiv.org/abs/1606.05250

    2016 arXiv

  29. [37]

    Unsupervised dense information retrieval with contrastive learning,

    G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave, “Unsupervised dense information retrieval with contrastive learning,” 2022. [Online]. Available: https://arxiv.org/abs/2112.09118

    2022 arXiv

  30. [38]

    Parameter- efficient fine-tuning of state space models,

    K. Galim, W. Kang, Y . Zeng, H. I. Koo, and K. Lee, “Parameter- efficient fine-tuning of state space models,” 2025. [Online]. Available: https://arxiv.org/abs/2410.09016

    2025 arXiv

  31. [39]

    Huggingface’s transformers: State-of-the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Huggingface’s transformers: State-...

    2020 arXiv

  32. [40]

    Quamba: A post-training quantization recipe for selective state space models,

    H.-Y . Chiang, C.-C. Chang, N. Frumkin, K.-C. Wu, and D. Marculescu, “Quamba: A post-training quantization recipe for selective state space models,” 2024. [Online]. Available: https://arxiv.org/abs/2410.13229

    2024 arXiv

  33. [41]

    Mambapeft: Exploring parameter-efficient fine-tuning for mamba,

    M. Yoshimura, T. Hayashi, and Y . Maeda, “Mambapeft: Exploring parameter-efficient fine-tuning for mamba,” 2025. [Online]. Available: https://arxiv.org/abs/2411.03855

    2025 arXiv

  34. [42]

    Mamba state-space models are lyapunov-stable learners,

    J. T. Halloran, M. Gulati, and P. F. Roysdon, “Mamba state-space models are lyapunov-stable learners,” 2025. [Online]. Available: https://arxiv.org/abs/2406.00209

    2025 arXiv

  35. [2021]

    Available: https://arxiv.org/abs/2106.09685

    [Online]. Available: https://arxiv.org/abs/2106.09685

    arXiv

  36. [2022]

    Available: https://arxiv.org/abs/2106.10199

    [Online]. Available: https://arxiv.org/abs/2106.10199

  37. [2024]

    Available: https://arxiv.org/abs/2404.13506

    [Online]. Available: https://arxiv.org/abs/2404.13506

    arXiv

  38. [2025]

    Available: https://arxiv.org/abs/2505.09388

    [Online]. Available: https://arxiv.org/abs/2505.09388

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.