SALE is a training-free sparse attention method that uses 4-bit quantized query-key estimates and a relative attention score to skip unimportant blocks, achieving over 3.36x prefill speedup on 64K+ token contexts with negligible accuracy loss.
Transformer Based Implementation for Automatic Book Summarization
1 Pith paper cite this work. Polarity classification is still indexing.
abstract
Document Summarization is the procedure of generating a meaningful and concise summary of a given document with the inclusion of relevant and topic-important points. There are two approaches: one is picking up the most relevant statements from the document itself and adding it to the Summary known as Extractive and the other is generating sentences for the Summary known as Abstractive Summarization. Training a machine learning model to perform tasks that are time-consuming or very difficult for humans to evaluate is a major challenge. Book Abstract generation is one of such complex tasks. Traditional machine learning models are getting modified with pre-trained transformers. Transformer based language models trained in a self-supervised fashion are gaining a lot of attention; when fine-tuned for Natural Language Processing(NLP) downstream task like text summarization. This work is an attempt to use Transformer based techniques for Abstract generation.
citation-role summary
citation-polarity summary
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
SALE is a training-free sparse attention method that uses 4-bit quantized query-key estimates and a relative attention score to skip unimportant blocks, achieving over 3.36x prefill speedup on 64K+ token contexts with negligible accuracy loss.