Pith. sign in

REVIEW 5 cited by

Streamlining Redundant Layers to Compress Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.19135 v5 pith:ORMQO5C3 submitted 2024-03-28 cs.CL cs.AI

Xiaodong Chen , Yuxuan Hu , Jing Zhang , Yanling Wang , Cuiping Li , Hong Chen This is my paper
classification cs.CLcs.AI
keywords layersllm-streamlinelayerpruninglanguagelargemetricmodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper introduces LLM-Streamline, a pioneer work on layer pruning for large language models (LLMs). It is based on the observation that different layers have varying impacts on hidden states, enabling the identification of less important layers to be pruned.LLM-Streamline comprises two parts: layer pruning, which removes consecutive layers with the lowest importance based on target sparsity, and layer replacement, a novel module that trains a lightweight network to replace the pruned layers to mitigate performance loss. Additionally, a new metric called stability is proposed to address the limitations of the widely used accuracy metric in evaluating model compression. Experiments show that LLM-Streamline outperforms both previous and concurrent state-of-the-art pruning methods in terms of both performance and training efficiency.Our code is available at https://github.com/RUCKBReasoning/LLM-Streamline

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A magnitude-compensation step that rescales the weights before a pruned layer, combined with iterative pruning, consistently improves perplexity and question-answering accuracy of layer-pruned LLMs.

  2. Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A task-conditioned projection operator converts a large transformer's weights into a smaller task-specialized transformer that outperforms same-size universal conditional models.

  3. A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Models

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    Tensor Mixture (MixT) compresses transformer linear layers into local tensor sums that execute directly, preserving MMLU accuracy up to a sharp per-model boundary and cutting LLaMA2-7B parameters by 47.5%.

  4. Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A training-free layer pruning framework for large vision-language models, combining token importance scoring with subspace-compensated weight projection, preserves most accuracy while speeding inference.

  5. SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling

    cs.CL 2025-06 conditional novelty 4.0 of 10

    SkipGPT learns per-token, per-module routers that dynamically skip attention and MLP blocks, then uses LoRA to recover most of the original model's quality.

Pith tools