Post-training loss of a pruned LLM follows a power-law curve fixed by pre-pruning model size, pruning rate, number of tokens, and the model's original loss.
$\rm SP^3$: Enhancing Structured Pruning via PCA Projection
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Structured pruning is a widely used technique for reducing the size of pre-trained language models (PLMs), but current methods often overlook the potential of compressing the hidden dimension (d) in PLMs, a dimension critical to model size and efficiency. This paper introduces a novel structured pruning approach, Structured Pruning with PCA Projection (SP3), targeting the effective reduction of d by projecting features into a space defined by principal components before masking. Extensive experiments on benchmarks (GLUE and SQuAD) show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, maintain over 96% accuracy, and outperform other methods that compress d by 6% in accuracy at the same compression ratio. SP3 has also proven effective with other models, including OPT and Llama. Our data and code are available at an anonymous repo.
fields
cs.AI 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
P$^2$ Law: Scaling Law for Post-Training After Model Pruning
Post-training loss of a pruned LLM follows a power-law curve fixed by pre-pruning model size, pruning rate, number of tokens, and the model's original loss.