Pith. sign in

REVIEW 6 major objections 5 minor 50 references

Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation

T0 review · 6 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey proposes a unified taxonomy of small language model design, compression, and evaluation, and assembles existing datasets into a platform for measuring SLM capabilities.

desk verdict A broad but sloppy survey of small LM techniques; the taxonomy is fine, but misattributed citations and an unvalidated invented metric make it unreliable as a field map. read the letter →

arxiv 2505.19529 v2 pith:ZBLM464Z submitted 2025-05-26 cs.CL

Tanjil Hasan Sakib , Md. Tanzib Hosain , Md. Kishor Morol This is my paper
classification cs.CL
keywords smalllanguagemodelsmodelcompressionpruningquantizationknowledgedistillationefficientinferenceedgedeploymentevaluationbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to be a complete guide to small language models, covering the architectures, training methods, and compression techniques that let compact models run in resource-limited settings. Its central contribution is a classification system that groups optimization approaches into pruning, quantization, and model compression, alongside categories for lightweight architecture, efficient attention, and knowledge transfer. It also assembles existing evaluation datasets and metrics into what it calls a rigorous platform for measuring SLM capabilities, organized by constraint such as latency, memory, privacy, and energy. A sympathetic reader would take the paper's aim to be giving researchers a way to choose and compare small models without reading the field paper by paper.

What carries the argument

The load-bearing device is the taxonomy presented across the paper's tables: Table 1 classifies SLMs by parameters, training tokens, token-to-parameter ratio, ALScore, architecture, and year; Table 2 divides training strategies into pre-training, fine-tuning, distributed training, and data augmentation; Table 3 divides compression into pruning, quantization, and knowledge transfer; Table 4 maps evaluation settings to constraints, datasets, and metrics. The taxonomy is what converts scattered model reports into a comparative guide, so the entire contribution stands on whether these categories and their examples are accurate.

What would settle it

Open the reference cited for Mamba, [Liu et al., 2024], and check whether it actually describes a state-space sequence model; open [Dao and Gu, 2024] and check whether it proposes Longformer; check whether the MobileLLM citation [Iu et al., 2024] resolves to the MobileLLM paper. If a substantial portion of such checks fails, the classification cannot serve as the map it claims to be.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in its abstract, is that the relevant knowledge about SLMs—design frameworks, training approaches, size- and complexity-reduction techniques, evaluation practices, and open problems—can be collected into a single assessment with a novel classification system. It argues that optimization approaches for SLMs fall into identifiable strategies (pruning, quantization, and model compression, plus training-side techniques), and that a constraint-oriented evaluation assembly can serve as a rigorous platform for measuring SLM capabilities. The paper supports this by tabulating existing models, methods, datasets, and metrics, and by identifying unresolved trade-offs between efficiency and performance.

Load-bearing premise

The guide is useful only if its citations point to the works it attributes them to; the text contains several places where this is not the case, so citation accuracy is the load-bearing premise.

Editorial extensions

If this is right

  • If the proposed classification holds, practitioners can navigate the SLM optimization landscape by strategy rather than by individual paper.
  • The assembled evaluation framework lets SLM capabilities be compared across efficiency, privacy, memory, and energy constraints.
  • The open-problems section identifies hallucination, bias, energy use, and privacy as the priorities that future work should target.
  • The taxonomy implies that compression is the main route for turning large pre-trained models into deployable small ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: If the taxonomy were corrected and re-verified against the original papers, it could serve as a shared index for comparing future SLMs, since its categories are general enough to absorb new methods.
  • Inference: The ALScore heuristic in Table 1 could be validated by correlating it with standard benchmark scores; a strong correlation would make it a cheap pre-training check, while a weak one would show it is only a rough ordinal scale.
  • Inference: The evaluation settings could be extended into standardized benchmarks with quantitative thresholds for latency, memory, privacy budget, and energy per token, which would make the proposed platform more directly actionable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. This paper is a survey of small language models (SLMs). It organizes the literature into architectural families (lightweight models, streamlined attention, neural architecture search, multimodal integration), training strategies (pre-training, fine-tuning, data augmentation), compression methods (pruning, quantization, knowledge distillation), evaluation datasets and metrics, and open problems such as hallucination, bias, energy efficiency, and privacy. The survey introduces a taxonomy of optimization approaches and proposes a quantity called ALScore, defined as sqrt(parameters × training tokens) / 300, which Table 1 uses as a rating of model power. The abstract claims a "complete assessment" of SLMs and a "rigorous platform" for measuring their capabilities.

Significance. If the survey were accurate, it could serve as a useful entry point for practitioners seeking an overview of SLM architectures and compression techniques. The paper covers a broad range of relevant topics and makes a reasonable organizational choice by separating training strategies from compression strategies. It also attempts to connect evaluation settings with concrete datasets and metrics. However, the paper's central value as a reliable field map is undermined by pervasive citation errors, an unvalidated invented metric used as evidence, tables that include models with 72B-176B parameters under the "small" label, and an evaluation section that lists components without providing a working evaluation protocol. These are not cosmetic issues: they affect the trustworthiness of the survey's core content. The paper offers no machine-checked proofs, reproducible code, or parameter-free derivations to offset these reliability problems.

major comments (6)
  1. [§2.2, references [Liu et al., 2024], [Dao and Gu, 2024]] The attribution of core architectures is incorrect in ways that misdirect readers. Mamba is attributed to [Liu et al., 2024], but the reference list entry is Mamba4Rec, a sequential recommendation paper, not the original Mamba model (Gu and Dao, 2023). Longformer is attributed to [Dao and Gu, 2024], which is the paper "Transformers are SSMs," not the Longformer paper. Since Section 2.2 is one of the main architecture classifications in the survey, these errors mean a reader who follows the citations will not find the described methods.
  2. [§4.2, reference [Han et al., 2015]] SpinQuant is attributed to [Han et al., 2015], but the cited reference is "Learning Both Weights and Connections for Efficient Neural Networks," an early pruning paper, not SpinQuant. This misattribution appears in the quantization section, where the reader is told that SpinQuant "turns outliers into a new space using rotation matrices," and the cited work does not support that claim.
  3. [§2.3 and reference [Iu et al., 2024]] MobileLLM is cited to [Iu et al., 2024] with the arXiv identifier 2401.12345, which is a placeholder and does not correspond to an identifiable paper. The actual MobileLLM work is by Liu et al. (2024). Additionally, in §2.1, BabyLLaMA and LLaMA are listed with the same arXiv identifier (2308.02019) in the reference list, making it impossible to locate the intended BabyLLaMA paper unambiguously.
  4. [§1 footnote 2 and Table 1] ALScore is introduced as a "quick and dirty rating of the model's power" with the formula sqrt(Parameters × Tokens) / 300 and a threshold of 1.0 for "powerful" models in mid-2023. No evidence or external benchmark is provided to justify this metric, yet Table 1 uses it as a quality indicator. This is an unvalidated, paper-specific metric presented as if it measured model capability, and it is disconnected from the evaluation metrics discussed in Section 5.
  5. [Table 1] Table 1 lists models with 72B (Qwen, Qwen2.5), 120B (Galactica), 175B (OPT), and 176B (BLOOM) parameters as examples of SLMs, with no stated parameter cutoff for what counts as "small." The "Tokens:Params" column is also inconsistently formatted: OPT's 300B tokens over 175B parameters is shown as "1:71" rather than as a comparable ratio such as 1.7:1. Several announcement years are inaccurate; for instance, OPT was released in 2022, not 2023. These issues undermine the survey's claim to provide a reliable comparative assessment.
  6. [§5, Table 4] The claim of assembling "a rigorous platform for measuring SLM capabilities" is not supported by the content of Section 5. The section presents a table of settings, datasets, and metrics, but it does not provide an evaluation protocol, an aggregation method, a validation of the platform against existing benchmarks, or a way to compare models. Listing datasets and metric names is not sufficient to constitute a measurement platform, and no evidence is given that the selected datasets and metrics are jointly usable for SLM assessment.
minor comments (5)
  1. [§1, footnote 2] The text says "Sqr Root" instead of "square root," and the ALScore definition would benefit from specifying units and from explaining why the divisor 300 is chosen.
  2. [§5.1] The text writes "n2c24" where it should be "n2c2," and the associated footnote points to the n2c2 project. The MIMIC dataset is mentioned without a citation.
  3. [Table 1] The "Tokens:Params" column should use a consistent numeric format; entries such as "1:71" are ambiguous and should be written as decimal ratios like "1.7:1."
  4. [Tables 2 and 3] The tables report quantitative performance gains (for example, "50% faster training," "60% size reduction," "retains 90% of teacher performance") without citing the sources for these specific numbers, making them unverifiable as presented.
  5. [Reference list] Several reference entries have incomplete author lists, inconsistent venue names, or appear to be invented or incorrect (for example, [Beyer and Jones, 2024], [Chen et al., 2023], [Laurencon and Picard, 2024], [Li and Zhao, 2024]). The entire reference list needs to be checked against the actual literature.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a survey with no derivation chain; the ALScore heuristic and citation misattributions are validity and reliability concerns, not circular reductions.

full rationale

The manuscript makes no formal derivations or first-principles predictions. Its core claims are a complete survey of SLMs, a proposed classification taxonomy, and an assembled evaluation suite built from existing datasets. These are organizational contributions assembled from external references rather than derived results. The ALScore metric is defined by the authors as sqrt(Parameters x Tokens)/300 and described as a quick rating of model power, but the paper only tabulates ALScore values in Table 1 and does not use the metric to predict or conclude model capability; no conclusion reduces to the ALScore definition by construction. Citation errors, such as attributing Mamba to the Mamba4Rec reference, Longformer to the Transformers are SSMs paper, SpinQuant to Han et al. 2015, and MobileLLM to a placeholder arXiv ID, are serious correctness problems for a survey's reliability, but they are not circularity under the required standard: they do not constitute a claim being justified by itself, by a fitted parameter renamed as a prediction, or by a load-bearing self-citation chain. No self-citations, imported uniqueness theorems, or ansatz-smuggling patterns are present. Because the hard rule requires exhibiting a specific equation-level or input-output reduction, and none exists here, the honest finding is no significant circularity with score 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The survey's central claims rest on the accuracy of the cited literature and on the paper's own selection of models and metrics. The misattributed references and the unvalidated ALScore metric are the main load-bearing assumptions that a reader would need to verify.

free parameters (2)
  • ALScore divisor = 300
    Chosen arbitrarily in Section 1 footnote 2; the threshold of 1.0 for 'powerful' is also hand-set with no calibration to any benchmark.
  • Small vs large cutoff = not stated
    The survey never defines a parameter boundary for 'small'; Table 1 includes 72B and 176B models, making the scope of the survey a selection choice.
assumptions (3)
  • domain assumption The cited works describe the methods attributed to them.
    The survey's usefulness depends on accurate citations. Section 2.2, 2.3, and 4.2 attribute Mamba, Longformer, SpinQuant, and MobileLLM to wrong or placeholder references.
  • domain assumption Table 1 parameter/token counts and ALScore values are accurate.
    The survey uses these values for comparisons but lists no per-row sources and the ALScore is unvalidated.
  • ad hoc to paper ALScore is a meaningful proxy for model power.
    Section 1 footnote 2 asserts this; no evidence links ALScore to task performance.
invented entities (1)
  • ALScore rating
    purpose: Quantify a language model's power from parameter count and training tokens
    Defined in Section 1 footnote 2 as sqrt(Params x Tokens)/300. It is an invented heuristic with no external validation, and the paper uses it as evidence in Table 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation." pith.science (2026). https://pith.science/paper/ZBLM464Z

@misc{pith2026250519529,
  author       = {Pith},
  title        = {Pith review of: Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZBLM464Z}},
  note         = {Machine review of arXiv:2505.19529}
}
read the original abstract

Small Language Models (SLMs) have gained substantial attention due to their ability to execute diverse language tasks successfully while using fewer computer resources. These models are particularly ideal for deployment in limited environments, such as mobile devices, on-device processing, and edge systems. In this study, we present a complete assessment of SLMs, focussing on their design frameworks, training approaches, and techniques for lowering model size and complexity. We offer a novel classification system to organize the optimization approaches applied for SLMs, encompassing strategies like pruning, quantization, and model compression. Furthermore, we assemble SLM's studies of evaluation suite with some existing datasets, establishing a rigorous platform for measuring SLM capabilities. Alongside this, we discuss the important difficulties that remain unresolved in this sector, including trade-offs between efficiency and performance, and we suggest directions for future study. We anticipate this study to serve as a beneficial guide for researchers and practitioners who aim to construct compact, efficient, and high-performing language models.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 20 canonical work pages

  1. [1]

    Paligemma: Towards compact vision encoders for multi- modal models.Transactions on Image Processing,

    [Beyer and Jones, 2024] Luc Beyer and Megan Jones. Paligemma: Towards compact vision encoders for multi- modal models.Transactions on Image Processing,

    work page 2024

  2. [3]

    In- ternvl2: Scalable multi-modal models with reduced vision encoder complexity

    [Chenet al., 2023 ] Jie Chen, Hao Wang, and Ming Zhang. In- ternvl2: Scalable multi-modal models with reduced vision encoder complexity. InAdvances in Neural Information Processing Systems,

    work page 2023

  3. [5]

    Flashattention: Fast and memory-efficient exact attention with io-awareness.arXiv preprint arXiv:2205.14135,

    [Daoet al., 2022 ] Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher R ´e. Flashattention: Fast and memory-efficient exact attention with io-awareness.arXiv preprint arXiv:2205.14135,

    arXiv 2022

  4. [7]

    Model tells you what to discard: Adaptive kv cache compression for llms

    [Geet al., 2023 ] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801,

    arXiv 2023

  5. [8]

    Realtoxic- ityprompts: Evaluating neural toxic degeneration in lan- guage models.arXiv preprint arXiv:2009.11462,

    [Gehmanet al., 2020 ] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxic- ityprompts: Evaluating neural toxic degeneration in lan- guage models.arXiv preprint arXiv:2009.11462,

    arXiv 2020

  6. [10]

    Hallusionbench: an advanced diagnostic suite for entangled language hallu- cination and visual illusion in large vision-language models

    [Guanet al., 2024 ] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallu- cination and visual illusion in large vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    work page 2024

  7. [11]

    Learning both weights and connections for efficient neural networks.Advances in Neural Infor- mation Processing Systems (NeurIPS), pages 1135–1143,

    [Hanet al., 2015 ] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks.Advances in Neural Infor- mation Processing Systems (NeurIPS), pages 1135–1143,

    work page 2015

  8. [13]

    [Havrilla, 2024] M. et al. Havrilla. Balancing noise and ac- curacy in privacy-preserving models.Proceedings of the ACM,

    work page 2024

Show all 50 references
  1. [14]

    Hooper and T

    [Hooper and Zhang, 2024] J. Hooper and T. Zhang. Key- value cache quantization for efficient long-sequence in- ference.Journal of Machine Learning Research,

    2024

  2. [15]

    The price of prompting: Profiling energy use in large language models inference

    [Husomet al., 2024 ] Erik Johannes Husom, Arda Goknil, Lwin Khin Shar, and Sagar Sen. The price of prompting: Profiling energy use in large language models inference. arXiv preprint arXiv:2407.16893,

    2024

  3. [16]

    Mobilellm: Efficient adaptation of large language models for mobile devices.ArXiv Preprint, abs/2401.12345,

    [Iuet al., 2024 ] Qiang Iu, Wei Zhang, Fang Chen, and Xi- aoyu Lin. Mobilellm: Efficient adaptation of large language models for mobile devices.ArXiv Preprint, abs/2401.12345,

    2024 arXiv

  4. [17]

    Tinybert: Distilling bert for natural language understanding

    [Jiaoet al., 2019 ] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351,

    2019 arXiv

  5. [19]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    [Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    2014 arXiv

  6. [21]

    Idefics2: Efficient multi-modal fusion with lightweight visual encoders.Transactions on Pattern Anal- ysis and Machine Intelligence,

    [Laurenc ¸on and Picard, 2024] Jean Laurenc ¸on and Thomas Picard. Idefics2: Efficient multi-modal fusion with lightweight visual encoders.Transactions on Pattern Anal- ysis and Machine Intelligence,

    2024

  7. [22]

    The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691,

    [Lesteret al., 2021 ] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691,

    2021 arXiv

  8. [24]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895,

    [Liet al., 2024a ] Feng Li, Renrui Zhang, Hao Zhang, Yuan- han Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895,

    arXiv

  9. [25]

    Llm-pbe: Assess- ing data privacy in large language models.arXiv preprint arXiv:2408.12787,

    [Liet al., 2024b ] Qinbin Li, Junyuan Hong, Chulin Xie, Jef- frey Tan, Rachel Xin, Junyi Hou, Xavier Yin, Zhun Wang, Dan Hendrycks, Zhangyang Wang, et al. Llm-pbe: Assess- ing data privacy in large language models.arXiv preprint arXiv:2408.12787,

    arXiv

  10. [26]

    Sophia: A memory-efficient optimizer for large-scale model training

    [Liu and Zhao, 2024] Fang Liu and Ting Zhao. Sophia: A memory-efficient optimizer for large-scale model training. Transactions on Machine Learning Research,

    2024

  11. [27]

    Prompt in- jection attack against llm-integrated applications.arXiv preprint arXiv:2306.05499,

    [Liuet al., 2023 ] Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. Prompt in- jection attack against llm-integrated applications.arXiv preprint arXiv:2306.05499,

    2023 arXiv

  12. [28]

    Mamba4rec: Towards efficient sequential recommendation with selective state space models.arXiv preprint arXiv:2403.03900,

    [Liuet al., 2024 ] Chengkai Liu, Jianghao Lin, Jianling Wang, Hanzhou Liu, and James Caverlee. Mamba4rec: Towards efficient sequential recommendation with selective state space models.arXiv preprint arXiv:2403.03900,

    2024 arXiv

  13. [29]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    [Loshchilov and Hutter, 2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    2019 arXiv

  14. [30]

    Mono-internvl: Mlp-based architectures for efficient multi-modal fusion

    [Luo and He, 2024] Qiang Luo and Yan He. Mono-internvl: Mlp-based architectures for efficient multi-modal fusion. Transactions on Neural Networks and Learning Systems,

    2024

  15. [31]

    Mixed pre- cision training

    [Micikeviciuset al., 2018 ] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed pre- cision training. InInternational Conference on Learning Representations,

    2018

  16. [33]

    Characterizing power management opportunities for llms in the cloud

    [Patelet al., 2024 ] Pratyush Patel, Esha Choukse, Chaojie Zhang, ´I˜nigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. Characterizing power management opportunities for llms in the cloud. InProceedings of the 29th ACM International Conference on Architect...

    2024

  17. [34]

    Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048,

    [Penget al., 2023 ] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048,

    2023 arXiv

  18. [35]

    Language models are unsupervised multitask learners

    [Radfordet al., 2019 ] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog,

    2019

  19. [36]

    Scaling language models: Methods, analysis & insights from training gopher.arXiv preprint arXiv:2112.11446,

    [Raeet al., 2021 ] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. Scaling language models: Methods, analysis & insights from training gopher.arXiv preprint arXiv:2...

    2021 arXiv

  20. [37]

    A primer in bertology: What we know about how bert works.Transactions of the Association for Com- putational Linguistics, 8:842–866,

    [Rogerset al., 2020 ] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works.Transactions of the Association for Com- putational Linguistics, 8:842–866,

    2020

  21. [38]

    Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    [Sanh, 2019] Victor Sanh. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    2019 arXiv

  22. [39]

    Towards greener llms: Bringing energy-efficiency to the forefront of llm inference.arXiv preprint arXiv:2403.20306,

    [Stojkovicet al., 2024a ] Jovan Stojkovic, Esha Choukse, Chaojie Zhang, Inigo Goiri, and Josep Torrellas. Towards greener llms: Bringing energy-efficiency to the forefront of llm inference.arXiv preprint arXiv:2403.20306,

    arXiv

  23. [40]

    Dy- namollm: Designing llm inference clusters for performance and energy efficiency.arXiv preprint arXiv:2408.00741,

    [Stojkovicet al., 2024b ] Jovan Stojkovic, Chaojie Zhang, ´I˜nigo Goiri, Josep Torrellas, and Esha Choukse. Dy- namollm: Designing llm inference clusters for performance and energy efficiency.arXiv preprint arXiv:2408.00741,

  24. [41]

    Mobilebert: A compact task-agnostic BERT for resource-limited devices

    [Sunet al., 2020 ] Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: A compact task-agnostic BERT for resource-limited devices. InProceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2158–217...

    2020

  25. [42]

    [Sunet al., 2023 ] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter

    Association for Computational Linguistics. [Sunet al., 2023 ] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695,

    2023 arXiv

  26. [44]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2308.02019,

    [Touvronet al., 2023 ] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- oth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Ham- bro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open an...

    2023 arXiv

  27. [45]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    [Xiaoet al., 2023 ] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. InInternational Conference on Machine Learning, pages 38087–38099. PMLR,

    2023

  28. [46]

    Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244,

    [Xuet al., 2023 ] Can Xu, Qingfeng Sun, Kai Zheng, Xi- ubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244,

    2023 arXiv

  29. [47]

    Privacy-preserving instructions for aligning large language models.arXiv preprint arXiv:2402.13659,

    [Yuet al., 2024 ] Da Yu, Peter Kairouz, Sewoong Oh, and Zheng Xu. Privacy-preserving instructions for aligning large language models.arXiv preprint arXiv:2402.13659,

    2024 arXiv

  30. [48]

    Llama-adapter: Efficient fine- tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199,

    [Zhanget al., 2023 ] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine- tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199,

    2023 arXiv

  31. [49]

    Tinyllama: An open-source small language model.arXiv preprint arXiv:2401.02385,

    [Zhanget al., 2024 ] Peiyuan Zhang, Guangtao Zeng, Tian- duo Wang, and Wei Lu. Tinyllama: An open-source small language model.arXiv preprint arXiv:2401.02385,

    2024 arXiv

  32. [50]

    Pytorch fsdp: experiences on scaling fully sharded data parallel

    [Zhaoet al., 2023 ] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277,

    2023 arXiv

  33. [51]

    Fanno: Augmenting high-quality instruction data with open-sourced llms only.arXiv preprint arXiv:2408.01323, 2024

    [Zhuet al., 2024 ] He Zhu, Junyou Su, Tianle Lun, Yicheng Tao, Wenjia Zhang, Zipei Fan, and Guanhua Chen. Fanno: Augmenting high-quality instruction data with open-sourced llms only.arXiv preprint arXiv:2408.01323, 2024

    2024 arXiv

  34. [2014]

    Reformer: The efficient transformer

    [Kitaevet al., 2020 ] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451,

    2020 arXiv

  35. [2015]

    Under- standing the effect of noise in llm training data with algo- rithmic chains of thought.arXiv preprint arXiv:2402.04004,

    [Havrilla and Iyer, 2024] Alex Havrilla and Maia Iyer. Under- standing the effect of noise in llm training data with algo- rithmic chains of thought.arXiv preprint arXiv:2402.04004,

    2024 arXiv

  36. [2018]

    Bbq: A hand-built bias benchmark for question answering.arXiv preprint arXiv:2110.08193,

    [Parrishet al., 2021 ] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thomp- son, Phu Mon Htut, and Samuel R Bowman. Bbq: A hand-built bias benchmark for question answering.arXiv preprint arXiv:2110.08193,

    2021 arXiv

  37. [2019]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    [Katharopouloset al., 2020 ] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InInternational conference on machine learning, pages 5156–5165. PMLR,

    2020

  38. [2020]

    Minillm: Knowledge distillation of large language models

    [Guet al., 2024 ] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InThe Twelfth International Conference on Learn- ing Representations,

    2024

  39. [2021]

    Mini-gemini: Efficient multi-modal models with lightweight vision en- coders.Proceedings of the International Conference on Machine Learning,

    [Li and Zhao, 2024] Peng Li and Hua Zhao. Mini-gemini: Efficient multi-modal models with lightweight vision en- coders.Proceedings of the International Conference on Machine Learning,

    2024

  40. [2022]

    Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323,

    [Frantaret al., 2022 ] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323,

    2022 arXiv

  41. [2023]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    [Dao and Gu, 2024] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, p...

    2024

  42. [2024]

    Retaining key information under high compression ra- tios: Query-guided compressor for llms.arXiv preprint arXiv:2406.02376,

    [Caoet al., 2024 ] Zhiwei Cao, Qian Cao, Yu Lu, Ningxin Peng, Luyang Huang, Shanbo Cheng, and Jinsong Su. Retaining key information under high compression ra- tios: Query-guided compressor for llms.arXiv preprint arXiv:2406.02376,

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.