A single dense-pretrained RAT+ model can be adapted with 1B tokens to run at dilation sizes D=2..128, matching dense accuracy at D=16 and losing only 1-3 points at D=64 on reasoning and long-context benchmarks, though retrieval tasks drop far more.
Rat: Bridging rnn efficiency and attention accuracy via chunk-based sequence modeling.arXiv preprint arXiv:2507.04416
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
years
2026 2representative citing papers
DLA introduces adaptive state merging based on token information variation plus fixed-size memory management for linear attention, reporting better results than fixed-policy baselines on 16 datasets across three categories.
citing papers explorer
-
RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference
A single dense-pretrained RAT+ model can be adapted with 1B tokens to run at dilation sizes D=2..128, matching dense accuracy at D=16 and losing only 1-3 points at D=64 on reasoning and long-context benchmarks, though retrieval tasks drop far more.
-
Dynamic Linear Attention
DLA introduces adaptive state merging based on token information variation plus fixed-size memory management for linear attention, reporting better results than fixed-policy baselines on 16 datasets across three categories.