pith. sign in

arxiv: 2110.15225 · v3 · pith:KUNAKWR6new · submitted 2021-10-28 · 💻 cs.CL · cs.LG

Pruning Attention Heads of Transformer Models Using A* Search: A Novel Approach to Compress Big NLP Architectures

Archit Parnami , Rahul Singh , Tarun Joshi This is my paper
classification 💻 cs.CL cs.LG
keywords modelstransformerattentionheadsaccuracyadoptionalgorithmbert
0
0 comments X
read the original abstract

Recent years have seen a growing adoption of Transformer models such as BERT in Natural Language Processing and even in Computer Vision. However, due to their size, there has been limited adoption of such models within resource-constrained computing environments. This paper proposes novel pruning algorithm to compress transformer models by eliminating redundant Attention Heads. We apply the A* search algorithm to obtain a pruned model with strict accuracy guarantees. Our results indicate that the method could eliminate as much as 40% of the attention heads in the BERT transformer model with no loss in accuracy.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Adaptive Head Budgeting for Efficient Multi-Head Attention

    cs.LG 2026-04 unverdicted novelty 7.0

    BudgetFormer adaptively budgets the number and selection of attention heads per input in Transformers, reducing FLOPs and memory on text classification while matching or exceeding standard multi-head performance.

  2. Adaptive Head Budgeting for Efficient Multi-Head Attention

    cs.LG 2026-04 conditional novelty 5.0

    BudgetFormer dynamically allocates a variable number of attention heads per input via a learned budget and relevance scoring, reducing inference cost on text classification while maintaining accuracy.