REVIEW 6 cited by
SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling
read the original abstract
Large language models (LLMs) achieve remarkable performance across tasks but incur substantial computational costs due to their deep, multi-layered architectures. Layer pruning has emerged as a strategy to alleviate these inefficiencies, but conventional static pruning methods overlook two critical dynamics inherent to LLM inference: (1) horizontal dynamics, where token-level heterogeneity demands context-aware pruning decisions, and (2) vertical dynamics, where the distinct functional roles of MLP and self-attention layers necessitate component-specific pruning policies. We introduce SkipGPT, a dynamic layer pruning framework designed to optimize computational resource allocation through two core innovations: (1) global token-aware routing to prioritize critical tokens, and (2) decoupled pruning policies for MLP and self-attention components. To mitigate training instability, we propose a two-stage optimization paradigm: first, a disentangled training phase that learns routing strategies via soft parameterization to avoid premature pruning decisions, followed by parameter-efficient LoRA fine-tuning to restore performance impacted by layer removal. Extensive experiments demonstrate that SkipGPT reduces over 40% of model parameters while matching or exceeding the performance of the original dense model across benchmarks. By harmonizing dynamic efficiency with preserved expressivity, SkipGPT advances the practical deployment of scalable, resource-aware LLMs. Our code is publicly available at: https://github.com/EIT-NLP/SkipGPT.
Forward citations
Cited by 6 Pith papers
-
Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models
Empirical analysis identifies task-interfering layers in VLMs and proposes TaLo, a test-time method to bypass them for improved performance without training.
-
End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference
L2A trains one LLM with input-and-budget-conditioned gates to adapt sparsity across layers, heads, and tokens, tracing the compute-accuracy frontier while staying within 0.6% of dense performance at 34% layer sparsity...
-
CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry
CascadeFormer tapers Transformer width with depth based on gradient fan-in asymmetry to match uniform baselines in perplexity while cutting latency 8.6%.
-
ProactiveLLM: Learning Active Interaction for Streaming Large Language Models
ProactiveLLM enables active interaction in streaming LLMs by learning semantic sufficiency cues from partial inputs through mask-based modeling and synchronized privileged self-distillation without external supervision.
-
SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation
An FPGA accelerator for SkipGPT-style pruned LLMs that fuses router/normalization/attention computation to hide nonlinear latency, packs two FP16 products in each DSP, and serves reused KV entries from on-chip URAM to...
-
ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention
Frozen visual tokens with sparse cross-attention at selected layers preserve 98% accuracy while reducing vision-side FLOPs to 4% in LLaVA-1.5 models.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.