{"work":{"id":"5d0e6adc-6dd2-49cc-8551-dc00433ed79f","openalex_id":"https://openalex.org/W4399694321","doi":"10.48550/arxiv.2406.06858","arxiv_id":"2406.06858","raw_key":null,"title":"FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion","authors":null,"authors_text":"Li-Wen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Chengji Yao, Ziheng Jiang, et al","year":2024,"venue":"cs.LG","abstract":"Large deep learning models have demonstrated strong ability to solve many tasks across a wide range of applications. Those large models typically require training and inference to be distributed. Tensor parallelism is a common technique partitioning computation of an operation or layer across devices to overcome the memory capacity limitation of a single processor, and/or to accelerate computation to meet a certain latency requirement. However, this kind of parallelism introduces additional communication that might contribute a significant portion of overall runtime. Thus limits scalability of this technique within a group of devices with high speed interconnects, such as GPUs with NVLinks in a node. This paper proposes a novel method, Flux, to significantly hide communication latencies with dependent computations for GPUs. Flux over-decomposes communication and computation operations into much finer-grained operations and further fuses them into a larger kernel to effectively hide communication without compromising kernel efficiency. Flux can potentially overlap up to 96% of communication given a fused kernel. Overall, it can achieve up to 1.24x speedups for training over Megatron-LM on a cluster of 128 GPUs with various GPU generations and interconnects, and up to 1.66x and 1.30x speedups for prefill and decoding inference over vLLM on a cluster with 8 GPUs with various GPU generations and interconnects.","external_url":"https://arxiv.org/abs/2406.06858","cited_by_count":3,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2406.06858","created_at":"2026-05-10T02:22:20.754805+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"Flux: Fast software-based communication over- lap on gpus through kernel fusion.arXiv preprint arXiv:2406.06858","render_title":"Flux: Fast software-based communication over- lap on gpus through kernel fusion.arXiv preprint arXiv:2406.06858"},"hub":{"state":{"work_id":"5d0e6adc-6dd2-49cc-8551-dc00433ed79f","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":20,"external_cited_by_count":3,"distinct_field_count":5,"first_pith_cited_at":"2025-05-11T17:28:30+00:00","last_pith_cited_at":"2026-07-08T18:54:08+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-22T17:59:46.044390+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":5},{"context_role":"method","n":2}],"polarity_counts":[{"context_polarity":"background","n":5},{"context_polarity":"use_method","n":2}],"runs":{},"summary":{},"graph":{},"authors":[]}}