GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers
Reviewed by Pithpith:V7HVNSIEopen to challenge →
read the original abstract
As transformers are equivariant to the permutation of input tokens, encoding the positional information of tokens is necessary for many tasks. However, since existing positional encoding schemes have been initially designed for NLP tasks, their suitability for vision tasks, which typically exhibit different structural properties in their data, is questionable. We argue that existing positional encoding schemes are suboptimal for 3D vision tasks, as they do not respect their underlying 3D geometric structure. Based on this hypothesis, we propose a geometry-aware attention mechanism that encodes the geometric structure of tokens as relative transformation determined by the geometric relationship between queries and key-value pairs. By evaluating on multiple novel view synthesis (NVS) datasets in the sparse wide-baseline multi-view setting, we show that our attention, called Geometric Transform Attention (GTA), improves learning efficiency and performance of state-of-the-art transformer-based NVS models without any additional learned parameters and only minor computational overhead.
This paper has not been read by Pith yet.
Forward citations
Cited by 9 Pith papers
-
The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups
Lie-Algebra Attention defines tokens as bare matrix Lie group elements and computes attention scores from the negative squared algebra norm of the relative pose log(g_i^{-1} g_j).
-
CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation
CRePE supplies depth-aware positional distributions along curved rays for stable unified-camera control in frozen video DiT models.
-
URoPE: Universal Relative Position Embedding across Geometric Spaces
URoPE is a parameter-free relative position embedding for transformers that works across arbitrary geometric spaces by ray sampling and projection, yielding consistent gains on novel view synthesis, 3D detection, trac...
-
WildSplat: Feedforward Gaussian Splatting from Unposed In-the-Wild Images
WildSplat decouples geometry from appearance in a single feedforward pass to produce appearance-conditioned 3D Gaussian reconstructions from unposed in-the-wild images.
-
DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers
DPPE decouples rotation and translation in camera positional encodings for multi-view transformers to resolve late-stage training stagnation and improve generalization in novel view synthesis.
-
Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis
StreetNVS presents a multi-sensor conditioned video diffusion framework for street-view novel view synthesis that outperforms baselines with sparse LiDAR and handles extreme out-of-trajectory paths on the Waymo dataset.
-
Memorize When Needed: Decoupled Memory Control for Spatially Consistent Long-Horizon Video Generation
A decoupled memory branch with hybrid cues, cross-attention, and gating improves spatial consistency and data efficiency in long-horizon camera-trajectory video generation.
-
DecMem: Towards Minute-Long Consistent World Generation with Decoupled Memory
DecMem proposes a decoupled memory system using sparse global and anchored local components to enable consistent minute-long controllable video generation in world models.
-
URoPE: Universal Relative Position Embedding across Geometric Spaces
RR2D reconstructs a virtual analog-domain covariance from partially observed hybrid array data via Toeplitz-constrained matrix completion, enabling practical hybrid SMI beamforming that outperforms direct hybrid SMI b...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.