Pith. sign in

REVIEW 6 cited by

Efficient Attentions for Long Document Summarization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2104.02112 v2 pith:O5RQ5RNL submitted 2021-04-05 cs.CL

Luyang Huang , Shuyang Cao , Nikolaus Parulian , Heng Ji , Lu Wang This is my paper
classification cs.CL
keywords efficientmodelsattentionsdocumentevaluationexistingheposlong
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The quadratic computational and memory complexities of large Transformers have limited their scalability for long document summarization. In this paper, we propose Hepos, a novel efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source. We further conduct a systematic study of existing efficient self-attentions. Combined with Hepos, we are able to process ten times more tokens than existing models that use full attentions. For evaluation, we present a new dataset, GovReport, with significantly longer documents and summaries. Results show that our models produce significantly higher ROUGE scores than competitive comparisons, including new state-of-the-art results on PubMed. Human evaluation also shows that our models generate more informative summaries with fewer unfaithful errors.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    Offline-learned head-reliability and risk-threshold tables make prefill-only KV compression recover about 97.7% of uncompressed LongBench accuracy at a 512-token-per-layer memory budget.

  2. EvolKV: Evolutionary KV Cache Compression for LLM Inference

    cs.LG 2025-09 conditional novelty 6.0 of 10

    CMA-ES search over per-layer KV cache budgets beats uniform and pyramidal compression heuristics on LongBench, NIAH, RULER, and GSM8K, and edges past the full cache on one code dataset at 1.5% of the budget.

  3. RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory

    cs.LG 2026-07 conditional novelty 5.0 of 10

    RED-PIM restructures attention to use a compact d×d intermediate matrix, reducing PIM inter-bank data movement from O(N²) to O(N) and claiming up to 99.99% latency reduction in simulation.

  4. From Global to Local: Learning Context-Aware Graph Representations for Document Classification and Summarization

    cs.CL 2026-02 conditional novelty 4.0 of 10

    Replacing full attention with sliding-window attention when building document sentence-graphs gives competitive classification accuracy at lower attention cost, with unfiltered local graphs best on medium-length news.

  5. HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models

    cs.CL 2025-09 reject novelty 4.0 of 10

    HoPE replaces RoPE's sine/cosine rotations with hyperbolic functions plus an exponential damping term to enforce monotonic attention decay, but the claimed consistent superiority and the 'RoPE as special case' theorem...

  6. PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    cs.LG 2025-09 conditional novelty 4.0 of 10

    PagedEviction prunes the KV cache in whole blocks using a key-value norm ratio, speeding up long-context LLM inference in vLLM while staying close to full-cache accuracy.

Pith tools