REVIEW 5 cited by
On the Effect of Dropping Layers of Pre-trained Transformer Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Transformer-based NLP models are trained using hundreds of millions or even billions of parameters, limiting their applicability in computationally constrained environments. While the number of parameters generally correlates with performance, it is not clear whether the entire network is required for a downstream task. Motivated by the recent work on pruning and distilling pre-trained models, we explore strategies to drop layers in pre-trained models, and observe the effect of pruning on downstream GLUE tasks. We were able to prune BERT, RoBERTa and XLNet models up to 40%, while maintaining up to 98% of their original performance. Additionally we show that our pruned models are on par with those built using knowledge distillation, both in terms of size and performance. Our experiments yield interesting observations such as, (i) the lower layers are most critical to maintain downstream task performance, (ii) some tasks such as paraphrase detection and sentence similarity are more robust to the dropping of layers, and (iii) models trained using a different objective function exhibit different learning patterns and w.r.t the layer dropping.
Forward citations
Cited by 5 Pith papers
-
Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders
Bekko a8m, with 7.7M active parameters, scores 56.2 on MMTEB Multilingual v2 Retrieval, beating mE5 models and BGE-M3, while a25m reaches 57.5, on par with gte-multilingual-base.
-
Staged Depth-Pruning Distillation of a Flow-Matching Text-to-Speech Teacher: A Compact Hindi Speech Synthesizer
Depth-only pruning of a flow-matching Hindi TTS teacher, followed by staged re-fine-tuning, produces 131–190M students with ASR-WER close to the teacher and real-time laptop inference.
-
PGB: One-Shot Pruning for BERT via Weight Grouping and Permutation
PGB prunes BERT in a single pass by permuting weight matrices into block-diagonal groups of important weights and removing all other weights, matching or beating iterative baselines on GLUE and SQuAD.
-
PrunePEFT: Iterative Hybrid Pruning for Parameter-Efficient Fine-tuning of LLMs
An iterative hybrid pruning method selects which PEFT modules to keep at each transformer layer, matching or improving fixed PEFT baselines on GLUE at 1% trainable parameters.
-
On Importance of Layer Pruning for Smaller BERT Models and Low Resource Languages
Layer-pruned MahaBERT-v2 and Google-Muril models roughly match full models on Marathi headline and paragraph classification but lose ground on document classification, and they do not always beat same-size scratch-tra...
Discussion (0). Continue with ORCID to comment.