Contribution Weights combine attention, value magnitude, and directional alignment to measure token influence more faithfully than attention alone, and show attention sinks actively suppress information via a convex sink-rate to output-norm relationship.
arXiv preprint arXiv:2408.10189 (2024)
6 Pith papers cite this work, alongside 3 external citations. Polarity classification is still indexing.
6
Pith papers citing it
3
external citations · external index
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
The paper casts the standard Transformer block with RoPE as a first-order approximation of a radial–tangential state estimator and introduces a Polar Transformer variant that retains the discarded geometric corrections.
HyLo upcycles Transformer LLMs into hybrids with MLA and Mamba2/Gated DeltaNet blocks via staged training and distillation, extending context to 2M tokens and outperforming prior upcycled hybrids on long-context benchmarks.
LightTransfer identifies lazy layers in LLMs like LLaMA and replaces their attention with streaming attention to form hybrid models, delivering up to 2.17x throughput with under 1.5% drop on LongBench and strong results on reasoning benchmarks.
PSCT-Net introduces a geometry-aware neural framework that uses differentiable back-projection and attention-guided 3D refinement to reconstruct pediatric skull CT from bi-planar X-rays.
citing papers explorer
-
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
Contribution Weights combine attention, value magnitude, and directional alignment to measure token influence more faithfully than attention alone, and show attention sinks actively suppress information via a convex sink-rate to output-norm relationship.
-
The Transformer as a Polar State Estimator
The paper casts the standard Transformer block with RoPE as a first-order approximation of a radial–tangential state estimator and introduces a Polar Transformer variant that retains the discarded geometric corrections.
-
Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
HyLo upcycles Transformer LLMs into hybrids with MLA and Mamba2/Gated DeltaNet blocks via staged training and distillation, extending context to 2M tokens and outperforming prior upcycled hybrids on long-context benchmarks.
-
LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation
LightTransfer identifies lazy layers in LLMs like LLaMA and replaces their attention with streaming attention to form hybrid models, delivering up to 2.17x throughput with under 1.5% drop on LongBench and strong results on reasoning benchmarks.
-
PSCT-Net: Geometry-Aware Pediatric Skull CT Reconstruction via Differentiable Back-Projection and Attention-Guided Refinement
PSCT-Net introduces a geometry-aware neural framework that uses differentiable back-projection and attention-guided 3D refinement to reconstruct pediatric skull CT from bi-planar X-rays.
- Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation