pith. sign in

arxiv: 2004.08483 · v5 · pith:ZCLFEMMQnew · submitted 2020-04-17 · 💻 cs.LG · stat.ML

ETC: Encoding Long and Structured Inputs in Transformers

Joshua Ainslie , Santiago Ontanon , Chris Alberti , Vaclav Cvicek , Zachary Fisher , Philip Pham , Anirudh Ravula , Sumit Sanghai
show 2 more authors
Qifan Wang Li Yang
This is my paper
classification 💻 cs.LG stat.ML
keywords inputsstructuredtransformerattentionencodingglobal-localinputlanguage
0
0 comments X
read the original abstract

Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks. In this paper, we present a new Transformer architecture, Extended Transformer Construction (ETC), that addresses two key challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs. To scale attention to longer inputs, we introduce a novel global-local attention mechanism between global tokens and regular input tokens. We also show that combining global-local attention with relative position encodings and a Contrastive Predictive Coding (CPC) pre-training objective allows ETC to encode structured inputs. We achieve state-of-the-art results on four natural language datasets requiring long and/or structured inputs.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MoBA: Mixture of Block Attention for Long-Context LLMs

    cs.LG 2025-02 unverdicted novelty 6.0

    MoBA routes attention over blocks via MoE-style gating to enable dynamic, bias-light long-context attention that matches full attention performance at lower cost.

  2. DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory

    cs.CV 2023-08 unverdicted novelty 6.0

    DragNUWA integrates text, image, and trajectory controls into a diffusion video model using a Trajectory Sampler, Multiscale Fusion, and Adaptive Training to enable fine-grained open-domain video generation.

  3. Deformable DETR: Deformable Transformers for End-to-End Object Detection

    cs.CV 2020-10 accept novelty 6.0

    Deformable DETR achieves higher accuracy than DETR, especially on small objects, while converging in one-tenth the training epochs by using sparse deformable attention on image features.