Pith. sign in

REVIEW 19 cited by

A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.03350 v2 pith:2RZJZRCD submitted 2024-11-04 cs.CL cs.AIcs.LG

Fali Wang , Zhiwei Zhang , Xianren Zhang , Zongyu Wu , Tzuhao Mo , Qiuhao Lu , Wanjing Wang , Rui Li
show 6 more authors
Junjie Xu Xianfeng Tang Qi He Yao Ma Ming Huang Suhang Wang
This is my paper
classification cs.CLcs.AIcs.LG
keywords modelsllmsslmslanguageacquisitionapplicationsknowledgelarge
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) have demonstrated emergent abilities in text generation, question answering, and reasoning, facilitating various tasks and domains. Despite their proficiency in various tasks, LLMs like PaLM 540B and Llama-3.1 405B face limitations due to large parameter sizes and computational demands, often requiring cloud API use which raises privacy concerns, limits real-time applications on edge devices, and increases fine-tuning costs. Additionally, LLMs often underperform in specialized domains such as healthcare and law due to insufficient domain-specific knowledge, necessitating specialized models. Therefore, Small Language Models (SLMs) are increasingly favored for their low inference latency, cost-effectiveness, efficient development, and easy customization and adaptability. These models are particularly well-suited for resource-limited environments and domain knowledge acquisition, addressing LLMs' challenges and proving ideal for applications that require localized data handling for privacy, minimal inference latency for efficiency, and domain knowledge acquisition through lightweight fine-tuning. The rising demand for SLMs has spurred extensive research and development. However, a comprehensive survey investigating issues related to the definition, acquisition, application, enhancement, and reliability of SLM remains lacking, prompting us to conduct a detailed survey on these topics. The definition of SLMs varies widely, thus to standardize, we propose defining SLMs by their capability to perform specialized tasks and suitability for resource-constrained settings, setting boundaries based on the minimal size for emergent abilities and the maximum size sustainable under resource constraints. For other aspects, we provide a taxonomy of relevant models/methods and develop general frameworks for each category to enhance and utilize SLMs effectively.

Discussion (0). Sign in to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Multi-Source Cybersecurity Logs: An ATT&CK-Labeled Dataset and SLM Evaluation

    cs.CR 2026-06 conditional novelty 8.0 of 10

    Introduces the first public multi-source cybersecurity log dataset with per-entry ATT&CK technique labels across 12 tactics and 53 techniques and demonstrates learnability via LoRA fine-tuning of three SLMs on chunk c...

  2. MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents

    cs.AI 2025-09 conditional novelty 7.0 of 10

    MAS-Bench introduces 139 tasks, 88 predefined shortcuts, and 9 metrics to evaluate hybrid GUI-shortcut mobile agents, reporting up to 68.3% success and 39% efficiency gains over GUI-only baselines.

  3. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

    cs.PF 2026-06 accept novelty 6.5 of 10

    On bandwidth-bound edge hardware, MoE inference cost tracks total parameters rather than active ones, so sparse activation fails to deliver the expected throughput or energy gains.

  4. Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Classification-head LoRA fine-tuning of sub-3B Qwen3 models outperforms label-generation SFT by 2–3% on HellaSwag, WinoGrande and PIQA and yields SOTA numbers competitive with GPT-3/PaLM/GPT-4.

  5. PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Existing MLLM unlearning methods reduce private-attribute leakage on entangled images but substantially harm co-occurring public figures and landmarks, with private knowledge often re-emerging after public finetuning.

  6. OrganicHAR: Towards Activity Discovery in Organic Settings for Privacy Preserving Sensors Using Efficient Video Analysis

    cs.HC 2026-05 unverdicted novelty 6.0 of 10

    OrganicHAR discovers 4-8 activity categories per user from sensor signals, achieves 79% accuracy on coarse activities with ambient sensors alone and cuts VLM queries by 90% by triggering video analysis only at detecte...

  7. Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A one-step belief propagation refinement with a repulsive Potts prior increases attention entropy and improves downstream accuracy of small Transformers, with GTD as a diagnostic for multi-hop attention.

  8. TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A small language model can rewrite cached large-model responses to fit similar new queries, preserving quality while cutting inference cost.

  9. Bradley-Terry and Multi-Objective Reward Modeling Are Complementary

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Jointly training a Bradley-Terry preference head and a multi-attribute regression head on a shared embedding improves reward-model robustness to reward hacking and boosts multi-objective scoring performance.

  10. Enhancing Linux Privilege Escalation Attack Capabilities of Local LLM Agents

    cs.CR 2026-04 unverdicted novelty 5.0 of 10

    Targeted prompting and system interventions enable local LLMs such as Llama 3.1 70B to exploit 83% of tested Linux privilege escalation vulnerabilities.

  11. Automatic Failure Attribution and Critical Step Prediction Method for Multi-Agent Systems Based on Causal Inference

    cs.AI 2025-09 reject novelty 5.0 of 10

    A hierarchical causal attribution framework (performance causal inversion, Shapley values, CDC-MAS step discovery) localizes failure agents and steps in LLM multi-agent systems, reporting up to 36.2% step accuracy.

  12. AI Propaganda factories with language models

    cs.CR 2025-08 conditional novelty 5.0 of 10

    Small language models sustain political personas and become more ideologically extreme when replying to counter-arguments, according to a language-model judge.

  13. Small Language Models are the Future of Agentic AI

    cs.AI 2025-06 unverdicted novelty 5.0 of 10

    Small language models are sufficiently capable, more suitable, and far more economical than large models for the repetitive tasks that dominate agentic AI systems.

  14. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

    cs.PF 2026-06 accept novelty 4.0 of 10

    Empirical benchmarks show MoE inference cost on edge hardware tracks total parameters rather than active parameters, with OLMoE-1B-7B behind dense baselines especially on the Jetson device.

  15. Large Language Models for Next-Generation Wireless Network Management: A Survey and Tutorial

    cs.NI 2025-09 conditional novelty 4.0 of 10

    A survey and tutorial that organizes LLM-enabled wireless network optimization into formulation, solution, and verification stages, with case studies drawn from the authors' own prior papers.

  16. Advancing SLM Tool-Use Capability using Reinforcement Learning

    cs.CL 2025-09 reject novelty 4.0 of 10

    GRPO with a strict reward for structured JSON output improves tool-call accuracy on small models, from 0.98%-6.1% to 22%-71% on the xLAM benchmark, though without baselines or variance.

  17. Unifying Mixture of Experts and Multi-Head Latent Attention for Efficient Language Models

    cs.AI 2025-08 reject novelty 4.0 of 10

    Combining MoE, MLA, and RoPE in small transformers improves perplexity on TinyStories while cutting KV cache size, but several headline numbers conflict internally.

  18. Large Language Model-Brained GUI Agents: A Survey

    cs.AI 2024-11 unverdicted novelty 4.0 of 10

    A survey consolidating frameworks, data practices, large action models, benchmarks, applications, and research gaps in LLM-brained GUI agents.

  19. Will LLMs Scaling Hit the Wall? Breaking Barriers via Distributed Resources on Massive Edge Devices

    cs.DC 2025-03 unverdicted novelty 2.0 of 10

    Position paper claiming that distributed training across massive edge devices can overcome data depletion and centralized compute monopolies in LLM scaling.

Pith tools