{"work":{"id":"65a18a30-6e80-4b64-a026-bb0368e38872","openalex_id":"https://openalex.org/W4389713725","doi":"10.48550/arxiv.2312.06635","arxiv_id":"2312.06635","raw_key":null,"title":"Gated Linear Attention Transformers with Hardware-Efficient Training","authors":null,"authors_text":"Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, Yoon Kim","year":2023,"venue":"cs.LG","abstract":"Transformers with linear attention allow for efficient parallel training but can simultaneously be formulated as an RNN with 2D (matrix-valued) hidden states, thus enjoying linear-time inference complexity. However, linear attention generally underperforms ordinary softmax attention. Moreover, current implementations of linear attention lack I/O-awareness and are thus slower than highly optimized implementations of softmax attention. This work describes a hardware-efficient algorithm for linear attention that trades off memory movement against parallelizability. The resulting implementation, dubbed FLASHLINEARATTENTION, is faster than FLASHATTENTION-2 (Dao, 2023) as a standalone layer even on short sequence lengths (e.g., 1K). We then generalize this algorithm to a more expressive variant of linear attention with data-dependent gates. When used as a replacement for the standard attention layer in Transformers, the resulting gated linear attention (GLA) Transformer is found to perform competitively against the LLaMA-architecture Transformer (Touvron et al., 2023) as well recent linear-time-inference baselines such as RetNet (Sun et al., 2023a) and Mamba (Gu & Dao, 2023) on moderate-scale language modeling experiments. GLA Transformer is especially effective at length generalization, enabling a model trained on 2K to generalize to sequences longer than 20K without significant perplexity degradations. For training speed, the GLA Transformer has higher throughput than a similarly-sized Mamba model.","external_url":"https://arxiv.org/abs/2312.06635","cited_by_count":5,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2312.06635","created_at":"2026-05-09T06:25:48.581424+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"Gated Linear Attention Transformers with Hardware-Efficient Training","render_title":"Gated Linear Attention Transformers with Hardware-Efficient Training"},"hub":{"state":{"work_id":"65a18a30-6e80-4b64-a026-bb0368e38872","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":50,"external_cited_by_count":5,"distinct_field_count":6,"first_pith_cited_at":"2024-01-18T17:55:39+00:00","last_pith_cited_at":"2026-07-09T01:15:03+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-23T07:09:39.932602+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":8},{"context_role":"method","n":3},{"context_role":"baseline","n":1}],"polarity_counts":[{"context_polarity":"background","n":7},{"context_polarity":"use_method","n":3},{"context_polarity":"baseline","n":1},{"context_polarity":"unclear","n":1}],"runs":{},"summary":{},"graph":{},"authors":[]}}