REVIEW 9 cited by
Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Our formulation reveals that the reduction across the sequence axis can be efficiently computed in parallel through a tree reduction. Our algorithm, called Tree Attention, for parallelizing exact attention computation across multiple GPUs enables cross-device decoding to be performed asymptotically faster (up to 8x faster in our experiments) than state-of-the-art approaches such as Ring Attention, while also requiring significantly less communication volume and incurring 2x less peak memory. We demonstrate that Tree Attention speeds up decoding up to 4x on Llama 3.1-8B and can be applied to a variety of hardware and networking setups such as H100 DGX nodes, AMD MI300x nodes, and PCIe connected NVIDIA RTX 4090s. Our code is publicly available here: https://github.com/Zyphra/tree_attention
Forward citations
Cited by 9 Pith papers
-
xGR: Efficient Generative Recommendation Serving at Scale
On real-world recommendation datasets, xGR sustains about 2.9–3.5× the throughput of vLLM/xLLM under a 200 ms P99 latency cap through GR-specific KV-cache, beam-search, and scheduling optimizations.
-
Pixel-Resolved Long-Context Learning for Turbulence at Exascale: Resolving Small-scale Eddies Toward the Viscous Limit
A multiscale transformer with a new collective-based parallel attention method is claimed to be the first deep-learning model to reproduce small-scale turbulence statistics down to the viscous limit in 3D flow.
-
NoLoCo: No-all-reduce Low Communication Training Method for Large Models
NoLoCo trains large language models without any all-to-all synchronization by using pairwise weight averaging and random pipeline routing, matching or slightly beating DiLoCo in experiments.
-
GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?
A new synthetic benchmark reveals that LLM reasoning accuracy decays sigmoidally with problem complexity and that repeated sampling has poor scaling efficiency.
-
BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference
BlockDialect assigns one of 16 FP4 'dialect' formats to each 32-64 element block of weights and activations, achieving near-full-precision accuracy with roughly 4-bit storage and integer-friendly MACs.
-
Star Attention: Efficient LLM Inference over Long Sequences
Star Attention speeds up long-context LLM inference by encoding context blocks locally with an anchor block, then computing global query attention, achieving near-parity accuracy with up to 11x speedup.
-
ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution
ZUNA1.1, an open-source 380M EEG diffusion autoencoder, reconstructs variable-length, flexibly masked EEG at least as well as its predecessor and far better than spherical spline interpolation.
-
TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication
TokenRing claims faster long-sequence attention on multiple GPUs by overlapping forward query transfers with backward partial-output transfers, supported by one 4-GPU profiling comparison.
-
A Survey on Large Language Model Acceleration based on KV Cache Management
A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.
Discussion (0). Continue with ORCID to comment.