{"work":{"id":"fff3953b-5efb-4753-bee4-002f59995810","openalex_id":"https://openalex.org/W4384648639","doi":"10.48550/arxiv.2307.08691","arxiv_id":"2307.08691","raw_key":null,"title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","authors":null,"authors_text":"Tri Dao","year":2023,"venue":"cs.LG","abstract":"Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in scaling to longer sequences, as its runtime and memory increase quadratically in the sequence length. FlashAttention exploits the asymmetric GPU memory hierarchy to bring significant memory saving (linear instead of quadratic) and runtime speedup (2-4$\\times$ compared to optimized baselines), with no approximation. However, FlashAttention is still not nearly as fast as optimized matrix-multiply (GEMM) operations, reaching only 25-40\\% of the theoretical maximum FLOPs/s. We observe that the inefficiency is due to suboptimal work partitioning between different thread blocks and warps on the GPU, causing either low-occupancy or unnecessary shared memory reads/writes. We propose FlashAttention-2, with better work partitioning to address these issues. In particular, we (1) tweak the algorithm to reduce the number of non-matmul FLOPs (2) parallelize the attention computation, even for a single head, across different thread blocks to increase occupancy, and (3) within each thread block, distribute the work between warps to reduce communication through shared memory. These yield around 2$\\times$ speedup compared to FlashAttention, reaching 50-73\\% of the theoretical maximum FLOPs/s on A100 and getting close to the efficiency of GEMM operations. We empirically validate that when used end-to-end to train GPT-style models, FlashAttention-2 reaches training speed of up to 225 TFLOPs/s per A100 GPU (72\\% model FLOPs utilization).","external_url":"https://arxiv.org/abs/2307.08691","cited_by_count":141,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2307.08691","created_at":"2026-05-09T06:00:33.632991+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","render_title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning"},"hub":{"state":{"work_id":"fff3953b-5efb-4753-bee4-002f59995810","tier":"super_hub","tier_reason":"100+ Pith inbound or 10,000+ external citations","pith_inbound_count":227,"external_cited_by_count":141,"distinct_field_count":23,"first_pith_cited_at":"2023-08-31T18:18:07+00:00","last_pith_cited_at":"2026-07-09T08:12:15+00:00","author_build_status":"needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-23T00:59:25.961582+00:00","tier_text":"super_hub"},"tier":"super_hub","role_counts":[{"context_role":"background","n":27},{"context_role":"method","n":20},{"context_role":"other","n":2},{"context_role":"baseline","n":1},{"context_role":"dataset","n":1}],"polarity_counts":[{"context_polarity":"background","n":27},{"context_polarity":"use_method","n":20},{"context_polarity":"unclear","n":2},{"context_polarity":"baseline","n":1},{"context_polarity":"use_dataset","n":1}],"runs":{"ask_index":{"job_type":"ask_index","status":"succeeded","result":{"title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","claims":[{"claim_text":"Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in scaling to longer sequences, as its runtime and memory increase quadratically in the sequence length. FlashAttention exploits the asymmetric GPU memory hierarchy to bring significant memory saving (linear instead of quadratic) and runtime speedup (2-4$\\times$ compared to optimized b","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"sentation Learning with PyTorch Geometric, arXiv:1903.02428 [cs] (Apr. 2019).doi:10. 48550/arXiv.1903.02428. URLhttp://arxiv.org/abs/1903.02428 [67] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Ré, FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, version Number: 2 (2022).doi:10.48550/ARXIV. 2205.14135. URLhttps://arxiv.org/abs/2205.14135 [68] T. Dao, FlashAttention-2: Faster Attention with Better Parallelism and Work Partition- ing, version Number: 1 (2023).doi:10. 48550/","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Current approaches largely assume that sink positions are static; however, sinks can emerge at non-initial positions depending on input and layer depth [28]. This introduces a fundamental trade-off: fixed-position methods are simple but may fail when sinks shift, whereas dynamic identification incurs additional computational overhead and can conflict with optimized kernels such asFlashAttention[188]. Efficient and accurate detection of dynamic sinks thus remains an open challenge. Future Directi","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"mance and even outperforms closed-source model RT-2-X in all categories except for semantic generalization. Average success rates ± StdErr are computed across 170 total rollouts per approach. See Table 4 for detailed results. techniques for large transformer model training such as automatic mixed precision (AMP, PyTorch [75]), FlashAttention [76], and fully sharded data parallelism (FSDP, Zhao et al. [77]). Out of the box, the OpenVLA codebase has full support for training on the Open X dataset,","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Vision and Pattern Recognition, 2024. [16] Errol Colak, Hui-Ming Lin, Robyn Ball, Melissa Davis, Adam Flanders, Sabeena Jalal, Kirti Magudia, Brett Marinelli, Savvas Nicolaou, Luciano Prevedello, Jeff Rudie, George Shih, Maryam Vazirabad, and John Mongan. Rsna 2023 abdominal trauma detection, 2023. URL https://kaggle.com/competitions/rsna-2023-abdominal-trauma-detection. [17] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691","claim_type":"background","confidence":0.7,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (3 contexts).","role_counts":[{"n":3,"context_role":"background"},{"n":1,"context_role":"method"}]},"error":null,"updated_at":"2026-05-15T13:37:48.210224+00:00"},"author_expand":{"job_type":"author_expand","status":"succeeded","result":{"authors_linked":[{"id":"fb143e2f-ed88-449e-b6c5-6527b627112b","orcid":null,"display_name":"Tri Dao"}]},"error":null,"updated_at":"2026-05-15T13:37:48.198356+00:00"},"context_extract":{"job_type":"context_extract","status":"succeeded","result":{"enqueued_papers":25},"error":null,"updated_at":"2026-05-14T06:37:23.732662+00:00"},"graph_features":{"job_type":"graph_features","status":"succeeded","result":{"co_cited":[{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":18},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":17},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":17},{"title":"Efficient Streaming Language Models with Attention Sinks","work_id":"a8d25452-c237-48c9-88a4-682717c3979a","shared_citers":15},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":12},{"title":"Longformer: The Long-Document Transformer","work_id":"abea7a44-6668-4de7-aab6-f53a6e5aa088","shared_citers":11},{"title":"Decoupled Weight Decay Regularization","work_id":"07ef7360-d385-4033-83f7-8384a6325204","shared_citers":9},{"title":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness","work_id":"efa96825-0830-4cfc-a250-fdaf6af302ab","shared_citers":9},{"title":"Rethinking Attention with Performers","work_id":"4c26d308-8b72-4a98-8e73-950617a75f50","shared_citers":9},{"title":"Fast Transformer Decoding: One Write-Head is All You Need","work_id":"160ea164-b1d4-4adb-8ccb-a4655d8a0bb4","shared_citers":8},{"title":"Linformer: Self-Attention with Linear Complexity","work_id":"4b717b51-6098-45d0-8e9e-b69bef651bc3","shared_citers":8},{"title":"LLaMA: Open and Efficient Foundation Language Models","work_id":"c018fc23-6f3f-4035-9d02-28a2173b2b9d","shared_citers":8},{"title":"PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling","work_id":"6317700d-f903-4ce1-8f53-b43cb146d48b","shared_citers":8},{"title":"RULER: What's the Real Context Size of Your Long-Context Language Models?","work_id":"c0bc4689-3ce8-4e3d-9442-bd74869445bb","shared_citers":8},{"title":"Training Verifiers to Solve Math Word Problems","work_id":"acab1aa8-b4d6-40e0-a3ee-25341701dca2","shared_citers":8},{"title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","work_id":"e96730e3-129b-4db6-b981-15ab7932e297","shared_citers":7},{"title":"arXiv preprint arXiv:2406.10774 , year=","work_id":"2cad64c9-e2d5-42b6-8db9-03fafde4bcb0","shared_citers":7},{"title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","work_id":"c5006563-f3ec-438a-9e35-b7b484f34828","shared_citers":7},{"title":"DeepSeek-V3 Technical Report","work_id":"57d2791d-2219-4c31-a077-afc04b12a75c","shared_citers":7},{"title":"Generating Long Sequences with Sparse Transformers","work_id":"c5b81688-45ee-4a9a-b095-e6290f45cb6c","shared_citers":7},{"title":"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints","work_id":"b73ad5b2-e553-4c71-b0c9-67e67ba7b158","shared_citers":7},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":7},{"title":"Qwen2.5 Technical Report","work_id":"d8432992-4980-4a81-85c7-9fa2c2b87f85","shared_citers":7},{"title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","work_id":"28ea1282-d657-4c61-a83c-f1249be6d6b1","shared_citers":7}],"time_series":[{"n":1,"year":2023},{"n":6,"year":2024},{"n":3,"year":2025},{"n":76,"year":2026}],"dependency_candidates":[]},"error":null,"updated_at":"2026-05-14T06:47:20.194917+00:00"},"identity_refresh":{"job_type":"identity_refresh","status":"succeeded","result":{"items":[{"title":"Qwen3 Technical Report","outcome":"unchanged","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","resolver":"local_arxiv","confidence":0.98,"old_work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e"}],"counts":{"fixed":0,"merged":0,"unchanged":1,"quarantined":0,"needs_external_resolution":0},"errors":[],"attempted":1},"error":null,"updated_at":"2026-05-14T06:37:27.833416+00:00"},"role_polarity":{"job_type":"role_polarity","status":"succeeded","result":{"title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","claims":[{"claim_text":"Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in scaling to longer sequences, as its runtime and memory increase quadratically in the sequence length. FlashAttention exploits the asymmetric GPU memory hierarchy to bring significant memory saving (linear instead of quadratic) and runtime speedup (2-4$\\times$ compared to optimized b","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"sentation Learning with PyTorch Geometric, arXiv:1903.02428 [cs] (Apr. 2019).doi:10. 48550/arXiv.1903.02428. URLhttp://arxiv.org/abs/1903.02428 [67] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Ré, FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, version Number: 2 (2022).doi:10.48550/ARXIV. 2205.14135. URLhttps://arxiv.org/abs/2205.14135 [68] T. Dao, FlashAttention-2: Faster Attention with Better Parallelism and Work Partition- ing, version Number: 1 (2023).doi:10. 48550/","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Current approaches largely assume that sink positions are static; however, sinks can emerge at non-initial positions depending on input and layer depth [28]. This introduces a fundamental trade-off: fixed-position methods are simple but may fail when sinks shift, whereas dynamic identification incurs additional computational overhead and can conflict with optimized kernels such asFlashAttention[188]. Efficient and accurate detection of dynamic sinks thus remains an open challenge. Future Directi","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"mance and even outperforms closed-source model RT-2-X in all categories except for semantic generalization. Average success rates ± StdErr are computed across 170 total rollouts per approach. See Table 4 for detailed results. techniques for large transformer model training such as automatic mixed precision (AMP, PyTorch [75]), FlashAttention [76], and fully sharded data parallelism (FSDP, Zhao et al. [77]). Out of the box, the OpenVLA codebase has full support for training on the Open X dataset,","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Vision and Pattern Recognition, 2024. [16] Errol Colak, Hui-Ming Lin, Robyn Ball, Melissa Davis, Adam Flanders, Sabeena Jalal, Kirti Magudia, Brett Marinelli, Savvas Nicolaou, Luciano Prevedello, Jeff Rudie, George Shih, Maryam Vazirabad, and John Mongan. Rsna 2023 abdominal trauma detection, 2023. URL https://kaggle.com/competitions/rsna-2023-abdominal-trauma-detection. [17] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691","claim_type":"background","confidence":0.7,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (3 contexts).","role_counts":[{"n":3,"context_role":"background"},{"n":1,"context_role":"method"}]},"error":null,"updated_at":"2026-05-15T13:37:48.205824+00:00"},"summary_claims":{"job_type":"summary_claims","status":"succeeded","result":{"title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","claims":[{"claim_text":"Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in scaling to longer sequences, as its runtime and memory increase quadratically in the sequence length. FlashAttention exploits the asymmetric GPU memory hierarchy to bring significant memory saving (linear instead of quadratic) and runtime speedup (2-4$\\times$ compared to optimized b","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-14T06:47:45.031369+00:00"}},"summary":{"title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","claims":[{"claim_text":"Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in scaling to longer sequences, as its runtime and memory increase quadratically in the sequence length. FlashAttention exploits the asymmetric GPU memory hierarchy to bring significant memory saving (linear instead of quadratic) and runtime speedup (2-4$\\times$ compared to optimized b","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning because it crossed a citation-hub threshold.","role_counts":[]},"graph":{"co_cited":[{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":18},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":17},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":17},{"title":"Efficient Streaming Language Models with Attention Sinks","work_id":"a8d25452-c237-48c9-88a4-682717c3979a","shared_citers":15},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":12},{"title":"Longformer: The Long-Document Transformer","work_id":"abea7a44-6668-4de7-aab6-f53a6e5aa088","shared_citers":11},{"title":"Decoupled Weight Decay Regularization","work_id":"07ef7360-d385-4033-83f7-8384a6325204","shared_citers":9},{"title":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness","work_id":"efa96825-0830-4cfc-a250-fdaf6af302ab","shared_citers":9},{"title":"Rethinking Attention with Performers","work_id":"4c26d308-8b72-4a98-8e73-950617a75f50","shared_citers":9},{"title":"Fast Transformer Decoding: One Write-Head is All You Need","work_id":"160ea164-b1d4-4adb-8ccb-a4655d8a0bb4","shared_citers":8},{"title":"Linformer: Self-Attention with Linear Complexity","work_id":"4b717b51-6098-45d0-8e9e-b69bef651bc3","shared_citers":8},{"title":"LLaMA: Open and Efficient Foundation Language Models","work_id":"c018fc23-6f3f-4035-9d02-28a2173b2b9d","shared_citers":8},{"title":"PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling","work_id":"6317700d-f903-4ce1-8f53-b43cb146d48b","shared_citers":8},{"title":"RULER: What's the Real Context Size of Your Long-Context Language Models?","work_id":"c0bc4689-3ce8-4e3d-9442-bd74869445bb","shared_citers":8},{"title":"Training Verifiers to Solve Math Word Problems","work_id":"acab1aa8-b4d6-40e0-a3ee-25341701dca2","shared_citers":8},{"title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","work_id":"e96730e3-129b-4db6-b981-15ab7932e297","shared_citers":7},{"title":"arXiv preprint arXiv:2406.10774 , year=","work_id":"2cad64c9-e2d5-42b6-8db9-03fafde4bcb0","shared_citers":7},{"title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","work_id":"c5006563-f3ec-438a-9e35-b7b484f34828","shared_citers":7},{"title":"DeepSeek-V3 Technical Report","work_id":"57d2791d-2219-4c31-a077-afc04b12a75c","shared_citers":7},{"title":"Generating Long Sequences with Sparse Transformers","work_id":"c5b81688-45ee-4a9a-b095-e6290f45cb6c","shared_citers":7},{"title":"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints","work_id":"b73ad5b2-e553-4c71-b0c9-67e67ba7b158","shared_citers":7},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":7},{"title":"Qwen2.5 Technical Report","work_id":"d8432992-4980-4a81-85c7-9fa2c2b87f85","shared_citers":7},{"title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","work_id":"28ea1282-d657-4c61-a83c-f1249be6d6b1","shared_citers":7}],"time_series":[{"n":1,"year":2023},{"n":6,"year":2024},{"n":3,"year":2025},{"n":76,"year":2026}],"dependency_candidates":[]},"authors":[{"id":"fb143e2f-ed88-449e-b6c5-6527b627112b","orcid":null,"display_name":"Tri Dao","source":"manual","import_confidence":0.72}]}}