Dual Triangle Attention achieves effective bidirectional attention with built-in positional inductive bias via dual triangular masks, outperforming standard bidirectional attention on position-sensitive tasks and showing strong masked language modeling results with or without positional embeddings.
Length generalization of causal transformers without position encoding
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
citation-role summary
method 1
citation-polarity summary
years
2026 2verdicts
UNVERDICTED 2roles
method 1polarities
use method 1representative citing papers
ATMA combines polar attention (direction + bounded-magnitude channels) with gated-delta recurrent compression to achieve length-invariant perplexity and >90% needle retrieval at 64K tokens after 2K training.
citing papers explorer
-
Dual Triangle Attention: Effective Bidirectional Attention Without Positional Embeddings
Dual Triangle Attention achieves effective bidirectional attention with built-in positional inductive bias via dual triangular masks, outperforming standard bidirectional attention on position-sensitive tasks and showing strong masked language modeling results with or without positional embeddings.
-
ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory
ATMA combines polar attention (direction + bounded-magnitude channels) with gated-delta recurrent compression to achieve length-invariant perplexity and >90% needle retrieval at 64K tokens after 2K training.