Pith. sign in

REVIEW 1 major objections 5 minor 9 cited by

RayRoPE encodes each image patch as a ray segment with predicted depth and uncertainty, which the paper shows makes multi-view attention SE(3)-invariant and geometry-adaptive and improves novel-view synthesis and stereo depth.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 08:56 UTC pith:BEAVHY7M

load-bearing objection Empirically compelling multi-view encoding with a fixable but real gap between its claimed relative-position property and the math as written. the 1 major comments →

arxiv 2601.15275 v3 pith:BEAVHY7M submitted 2026-01-21 cs.CV cs.LG

RayRoPE: Projective Ray Positional Encoding for Multi-view Attention

Yu Wu , Minsik Jeon , Jen-Hao Rick Chang , Oncel Tuzel , Shubham Tulsiani This is my paper
classification cs.CV cs.LG
keywords positional encodingrotary position embeddingmulti-view attentionnovel view synthesisstereo depth estimationSE(3) invarianceray representationuncertainty modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multi-view transformers need to know where each image patch is in 3D, but existing positional encodings tend to fail at least one of four basic requirements: invariance to the choice of world coordinate system, unique encoding of a patch across views, adaptation to scene geometry, and multi-frequency similarity. RayRoPE addresses all four by representing each patch as a ray segment—camera center plus a point at a per-token predicted depth—projecting every ray into the query token's camera frame, and applying a rotary encoding that is averaged analytically over a predicted depth uncertainty interval. The paper's empirical claim is that this encoding consistently outperforms prior absolute and relative position encodings for novel-view synthesis and stereo depth estimation, with a 24% relative LPIPS improvement reported on one real-estate benchmark and gains that grow with more reference views. If the claim holds, geometry can be injected into attention without explicit supervision, and attention becomes a more reliable spatial reasoning mechanism for 3D vision.

Core claim

On the paper's own terms, the central discovery is that ray-based, query-frame, uncertainty-damped rotary encoding satisfies the four desiderata simultaneously. Each token predicts a depth d and uncertainty σ via linear layers trained without direct supervision; those turn a ray into a segment [p_(d−σ), p_(d+σ)]. Projecting all segments into the query camera frame makes the encoding depend only on the relative projected positions, hence invariant to rigid world transformations, while the closed-form expectation E[e^(iωx)] = sinc(ωΔ/2)e^(iωm) damps high frequencies when depth is uncertain. The paper validates this on novel-view synthesis and stereo depth estimation and reports that deeper lay

What carries the argument

The central object is a ray segment x=(c, p_d) in homogeneous coordinates: c is the camera center and p_d is the point at predicted depth d along the patch's ray, with uncertainty σ widening it to a uniform segment. Projection π(P_i,x)=(T_i c, K_i T_i p_d) into the query camera gives a 6D projective position. The encoding is the analytic expectation of the standard rotary map—ordinary sinc at ωΔ/2 times a rotation by the segment midpoint—applied blockwise over multiple frequencies. This single construction carries all four desiderata: projection gives SE(3) invariance, the ray gives uniqueness, predicted depth gives geometry adaptivity, and multi-frequency blocks give coarse-to-fine similari

Load-bearing premise

The load-bearing premise is that attention scores reduce to a function of relative projected ray positions only (Eq. 7 and Eq. 10); that reduction uses the expectation of the negative-angle encoding for the key side (Appendix A.2, Eq. 11), a convention left implicit in the main text, so under the literal matrix inverse of Eq. 6 the invariance claim would fail.

What would settle it

At a single frequency, compute attention weights two ways for a query with fixed position and two keys with the same midpoint and same relative displacement but different uncertainties Δ: once using Eq. 6 with the literal matrix inverse and once using Eq. 7. If the softmax ranking changes—i.e., if the ratio sinc(ωΔ_i/2)/sinc(ωΔ_j/2) appears—the relative-position-only claim is false; the same check can be applied end-to-end by freezing depths and uncertainties to constants and seeing whether RayRoPE's gains disappear.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Novel-view synthesis improves across three benchmarks and two model scales, with the largest reported gain being a 24% relative LPIPS improvement on one real-estate benchmark.
  • Stereo depth estimation improves on indoor benchmarks and on an unseen test domain, indicating the encoding's benefit transfers beyond the training distribution.
  • When ground-truth depth is available for reference views, substituting it into the ray representation and zeroing uncertainty yields larger synthesis gains, giving known geometry a direct route into attention.
  • Removing any one component—uncertainty modeling, depth prediction, multi-frequency blocks, or value/output encoding—degrades results, so all four design elements are load-bearing in the reported regime.
  • The predicted depth is unsupervised yet meaningful: by deeper layers it becomes geometrically plausible, and per-token uncertainty tracks depth error, suggesting attention can discover scene structure without depth labels.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The expected-encoding trick is general: any positional feature with a bounded interval of uncertainty—not just depth along a ray—can be turned into a damped rotary encoding, which could transfer to track-based or flow-based position encodings where positions are noisy.
  • Because depth and uncertainty are predicted per token per layer, RayRoPE can adapt its geometric locality at different abstraction levels; one testable extension is that increasing frequency bands should matter more on scenes with fine parallax than on distant-view datasets.
  • A stricter implementation check: if the key-side inverse uses the literal matrix inverse rather than the expectation of the negative-angle encoding, attention scores gain an uncertainty ratio; comparing those two variants would isolate whether the reported gains come from the relative-position property or from uncertainty-based reweighting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes RayRoPE, a relative positional encoding for multi-view transformers. Each patch token is represented as a ray segment whose depth is predicted per token (along with an uncertainty), and the ray is projected into the query camera frame. The authors derive an expected RoPE under a uniform depth-uncertainty distribution and claim the resulting attention depends only on relative projected positions, is SE(3)-invariant, multi-frequency, and geometry-adaptive. They validate the method by integrating it into LVSM, EscherNet, and UniMatch, reporting consistent improvements on novel-view synthesis and stereo depth estimation across multiple datasets.

Significance. If the derivation is corrected, RayRoPE would be a genuinely useful contribution: it provides a principled way to combine SE(3)-invariant relative encoding with multi-frequency RoPE and geometry-aware position adaptation. The empirical evaluation is extensive and consistent: Tables 1–3 show RayRoPE outperforming prior positional encodings on three datasets and three backbones, and the ablations in Table 4 support the importance of depth, uncertainty, and multi-frequency components. The paper also demonstrates generalization to varying numbers of reference views and out-of-domain scenes. However, the central theoretical claim—that the expected encoding preserves a pure relative-position dependency—is not established as written, which is a load-bearing issue that must be fixed before the paper can be accepted.

major comments (1)
  1. [§4.3, Eq. (10); Algorithm 1; §A.2, Eq. (11)] The claimed identity E[ρ(x_i)](E[ρ(x_j)])^{-1}=E[ρ(x_i-x_j)] is false with the damped expected encoding of Eqs. (8)–(9). Writing E[ρ(x)]=S(x)R(μ(x)) with S=sinc(ωΔ/2), the left side equals (S_i/S_j)R(μ_i-μ_j), whereas the right side equals S_i S_j R(μ_i-μ_j). The proof in §A.2 (Eq. 11) uses E[ρ(x_j)^{-1}], the expectation of the negative-angle encoding, which differs from (E[ρ(x_j)])^{-1} unless S_j=1. Algorithm 1's Enc^{-1} denotes the matrix inverse, so the implementation description contradicts the proof. Consequently, Eq. (7)'s relative-position-only form and the SE(3)-invariance argument are not established as written. Please correct Eq. (10) and Algorithm 1 to use the expected negative-angle encoding on the key/value side, or, if the literal inverse is used, state this explicitly and analyze the resulting per-token uncertainty ratio S_i/S_j.
minor comments (5)
  1. [Abstract] The '24% relative improvement on LPIPS in RE10K' is computed relative to the best baseline (PRoPE), not to the worst baseline in Table 1. Please state the reference method for these relative numbers.
  2. [§2] Typo: 'multivew' should be 'multi-view'.
  3. [Algorithm 1] The notation 'Enc^{-1}K' is ambiguous; use explicit matrix multiplication and clarify that the inverse is taken per 2x2 block when applying the encoding.
  4. [§5.1] '8× batch size' does not specify the effective batch size; report the exact number for the large models.
  5. [§A.2] The proof in the appendix should be labeled as proving a corrected version of Eq. (10), and the relationship between E[ρ(x)^{-1}] and (E[ρ(x)])^{-1} should be explicitly noted.

Circularity Check

0 steps flagged

No meaningful circularity: the RayRoPE derivation is self-contained and empirically anchored on held-out benchmarks; the only self-citation is a minor, non-load-bearing CO3D split reference.

full rationale

RayRoPE's derivation chain is not circular. The position representation x=(c,p_d) is defined from camera geometry and a per-token depth predicted by a linear layer; the expected RoPE (Eqs. 8-9) is an analytic Fourier expectation over a uniform ray segment, computed in closed form. The relative-position claim (Eqs. 6-7, 10) is a mathematical identity about RoPE rotations; it does not take its conclusion as an input, and any ambiguity about whether 'inverse' is taken before or after the expectation (as highlighted by the skeptic) is a correctness concern, not a circularity. The depth d and uncertainty sigma are unsupervised latent variables learned jointly with the model, not parameters fitted to the evaluation metrics. The empirical validation is performed on held-out external benchmarks (RE10K, CO3D, Objaverse, DL3DV, ScanNet), so the reported improvements are not forced by construction. The only author-overlapping citation is [48] (Cameras as Rays), used solely to define the CO3D train/eval split in Appendix B ('For CO3D [31], we follow the split in [48].'); this is a minor dataset-protocol reference and is not load-bearing for RayRoPE's derivation or for the reported gains. No step in the paper reduces by definition to its inputs, and no fitted quantity is renamed as a prediction. Score 2 reflects the single non-load-bearing self-citation; the central content is independent.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

RayRoPE's central claims rest on: (i) the standard math of RoPE rotations; (ii) the projective camera model; (iii) the unsupervised learnability of per-token depth and uncertainty - empirically supported by ablations and depth-error/uncertainty correlations, but not proven; (iv) stated modeling approximations (componentwise uniform projected positions, independence across tokens in the expected RoPE); and (v) an unstated convention for inverting damped expected encodings, which the paper's notation and its appendix proof do not pin down. No new physical entities are introduced: depth and uncertainty are latent variables predicted from features, not independent postulates with external falsifiable handles.

free parameters (4)
  • Per-token depth d (per attention layer) = learned per-token (linear layer W_d, exp-activated); no direct supervision
    The geometry-adaptiveness of RayRoPE (Section 4.1) is carried entirely by this unsupervised prediction; ablation (2) confirms it matters (CO3D PSNR 18.40->17.06 and RE10K 26.07->25.36 when removed). Its geometric validity is verified only empirically and only in-distribution (Fig. 6).
  • Per-token uncertainty sigma (per attention layer) = learned per-token (linear layer W_sigma, exp-activated)
    Controls the damping/smoothing of the expected RoPE (Section 4.3); ablation (1) shows a large CO3D drop (18.40->17.28 PSNR, 0.461->0.594 LPIPS).
  • RoPE frequency schedule and per-component scaling for the 6D projected ray = unspecified in the paper
    The 6D vector mixes world coordinates, pixel coordinates, and disparity (Section 4.2 Eq. 5). Standard RoPE frequencies are described as 'predefined' (Section 3.1) but the paper never states how they are scaled across these heterogeneous units - a hand-chosen quantity that plausibly governs the multi-frequency similarity behavior.
  • Number of rays per patch (3, through patch corners) = 3
    Design choice stated in Section 4.1 to encode patch orientation; the combination rule (concatenation vs averaging) is not described.
axioms (6)
  • standard math Rotary position encoding satisfies rho(a) rho(b) = rho(a+b) and rho(x)^{-1} = rho(-x) (used in Eqs. 6-7 to reduce attention to relative positions)
    Invoked in Section 3.1 and used in the derivation of Eq. 7; true for pure rotations, only approximately true for the damped expected encodings of Eq. 9.
  • domain assumption Projective camera model P_i = K_i T_i with homogeneous ray/point representation (c, p_d)
    Standard CV; used in Eq. 5 to define query-frame projected rays. Not proven, but standard.
  • domain assumption Token features (and hence predicted d, sigma) are SE(3)-invariant functions of the inputs; global frame changes do not affect images
    Needed for the SE(3)-invariance claim of Section 4.2; the authors also normalize poses so the first view's extrinsics are identity (Section B).
  • ad hoc to paper Projected positions follow a uniform distribution componentwise on [x_min, x_max] (Eq. 8-9)
    Stated modeling approximation in Section 4.3; the true distribution of projected points under depth uncertainty is not uniform, especially for the disparity component (1/d').
  • ad hoc to paper Independence of x_i and x_j in the expected-RoPE factorization (Eq. 10; App. Eq. 11)
    Assumed so the expectation factorizes; reasonable for tokens from different views, less so for overlapping or adjacent tokens.
  • ad hoc to paper The key-side inverse encoding in Eq. 6 is interpreted as E[rho(x_j)^{-1}] (expectation of the negative-angle encoding) rather than (E[rho(x_j)])^{-1} (matrix inverse of the damped matrix)
    Required for Eq. 7 and Eq. 10 to hold exactly; the paper's notation rho(x_j_tilde)^{-1} suggests the matrix inverse, and the implementation convention is never stated.

pith-pipeline@v1.3.0-alltime-deepseek · 5314 in / 5575 out tokens · 384671 ms · 2026-08-03T08:56:37.839760+00:00 · methodology

0 comments
read the original abstract

We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can adapt to the geometry of the underlying 3D scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet these desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays and computes query-frame projective coordinates to ensure SE(3) invariance. To adapt to scene geometry, RayRoPE predicts (without direct supervision) a per-token depth to obtain its position along the corresponding ray, while also modeling uncertainty and analytically computing the expected positional encoding. We validate our method on the tasks of novel-view synthesis, stereo depth estimation, and feed-forward 3DGS reconstruction. While remaining efficient, RayRoPE consistently improves over alternate position encoding schemes (e.g., 24% relative improvement on LPIPS in RE10K).

Figures

Figures reproduced from arXiv: 2601.15275 by Jen-Hao Rick Chang, Minsik Jeon, Oncel Tuzel, Shubham Tulsiani, Yu Wu.

Figure 1
Figure 1. Figure 1: Desiderata for a Multi-View Position Encoding. We seek the following properties for position encoding for multi-view attention: (a) The attention output should be invariant to the choice of global coordinate, namely SE(3) invariance. (b) The positional encoding of tokens that correspond to the same patch observed across different images should be the same. (c) The positional encoding can vary with the unde… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of RayRoPE. (a) We encode image patch position as a ray segment x = (c, p d ), where c is the camera center and p d is the point at depth d along the ray r. We use a linear layer to allow each token to predict the depth d along the ray, thus enabling RayRoPE to adapt to the scene geometry. (b) To ensure SE(3) invariance, we compute the positional encodings using ray positions projected to the quer… view at source ↗
Figure 3
Figure 3. Figure 3: Applying RayRoPE to the Attention Layer. We apply a linear layer on features τ to predict a per-token depth, which is used to compute ray segments. When a query token τi attends to a set of key tokens {τj}, we compute positional encoding ρD by projecting rays with query camera Pi. We apply the RayRoPE en￾coding to q, k, v and o features. subsequent discussion. 4.2. Relative Encodings in Query Frame The ray… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative examples on novel view synthesis. RayRoPE synthesizes more 3D-consistent views with sharper details [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative examples on stereo depth estimation. We visualize reprojected 3D points from stereo depth estimation results. When applied to UniMatch [45], RayRoPE leads to more accurate depth predictions, resulting in improved 3D reconstruction quality. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Left: Error on predicted depths vs predicted uncertainties. In deeper layers (Layer 5–6), the depth errors and uncertainties are strongly positively correlated, demonstrating that the model predicted higher uncertainty for depth with lower confidence. Right: Predicted depths and uncertainties across layers. The predicted depth at deeper layers aligns well with the ground-truth depth. The predicted σ gradua… view at source ↗
Figure 7
Figure 7. Figure 7: Out-of-distribution generalization Left: Evaluating on varying numbers of reference views (trained only with two reference views). Right: Evaluating models trained on CO3D on out-of-distribution categories and datasets. RayRoPE maintains its advantage in both settings. Full results are in the appendix. 5.4. NVS with Known Depth To further highlight RayRoPE’s adaptiveness to scene ge￾ometry, we train and ev… view at source ↗
Figure 8
Figure 8. Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Analysis on attention similarities as ray position varies. We plot attention similarities between two patches as their relative position varies in three settings. The top row illustrates each setting from a top-down view: (a) The cameras are translated by ∆x. (b) The cameras are rotated by ∆θ (c) Both patches observe the same 3D point at depth 1. We fix the predicted depth for the first patch d1 to 1, whil… view at source ↗
Figure 10
Figure 10. Figure 10: Effect of varying the number of reference views. Qualitative results with different numbers of reference images. Method CO3D Unseen Categories Objaverse RE10K PSNR↑ LPIPS↓ SSIM↑ PSNR↑ LPIPS↓ SSIM↑ PSNR↑ LPIPS↓ SSIM↑ Plucker raymap 17.08 0.639 0.579 10.81 0.523 0.714 15.78 0.680 0.493 RoPE on rays 17.57 0.593 0.560 14.22 0.394 0.762 19.00 0.437 0.578 GTA [26] 16.99 0.576 0.585 14.44 0.342 0.753 18.36 0.460… view at source ↗
Figure 11
Figure 11. Figure 11: Qualitative Results on EscherNet. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Additional predicted depth visualizations. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Additional examples from CO3D. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Additional examples from RE10K. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Additional examples from Objaverse. The top three rows show target views with radial variations only, while the bottom three rows show target views with compound variations. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation

    cs.CV 2026-05 unverdicted novelty 7.0

    CRePE supplies depth-aware positional distributions along curved rays for stable unified-camera control in frozen video DiT models.

  2. URoPE: Universal Relative Position Embedding across Geometric Spaces

    cs.CV 2026-04 unverdicted novelty 7.0

    URoPE is a parameter-free relative position embedding for transformers that works across arbitrary geometric spaces by ray sampling and projection, yielding consistent gains on novel view synthesis, 3D detection, trac...

  3. EO-VGGT: Orbital Ray-Conditioned 3D Foundation Models for Satellite Multi-View Reconstruction

    cs.CV 2026-07 unverdicted novelty 6.0

    EO-VGGT introduces geometry-constrained view selection, a sensor-ray encoder, and a ray-pointing adapter to enable feed-forward 3D reconstruction from orbital satellite images.

  4. DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers

    cs.CV 2026-06 unverdicted novelty 6.0

    DPPE decouples rotation and translation in camera positional encodings for multi-view transformers to resolve late-stage training stagnation and improve generalization in novel view synthesis.

  5. RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways

    cs.LG 2026-06 unverdicted novelty 6.0

    RoVE rotates value embeddings simultaneously with keys in attention to make values position-dependent, reframing RoPE as attentive convolution and reporting gains on long-context tasks in 124M and 354M GPT-2 models.

  6. Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis

    cs.CV 2026-06 unverdicted novelty 6.0

    StreetNVS presents a multi-sensor conditioned video diffusion framework for street-view novel view synthesis that outperforms baselines with sparse LiDAR and handles extreme out-of-trajectory paths on the Waymo dataset.

  7. PE-Field 4D: Video Generation Models as Canvas

    cs.CV 2026-07 conditional novelty 5.0

    Warping reference tokens' positional encodings into the target view, with depth offsets and frame-level compression fixes, improves geometry-aware camera control in video diffusion transformers.

  8. RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways

    cs.LG 2026-06 conditional novelty 5.0

    Rotating value embeddings along with keys and queries (RoVE) converts RoPE attention into a block-Toeplitz attentive convolution and yields consistent gains across 124M and 354M GPT-2 models.

  9. URoPE: Universal Relative Position Embedding across Geometric Spaces

    cs.CV 2026-04 conditional novelty 5.0

    RR2D reconstructs a virtual analog-domain covariance from partially observed hybrid array data via Toeplitz-constrained matrix completion, enabling practical hybrid SMI beamforming that outperforms direct hybrid SMI b...

Reference graph

Works this paper leans on

52 extracted references · 6 linked inside Pith · cited by 7 Pith papers

  1. [1]

    Ac3d: Analyzing and improving 3d camera control in video diffusion trans- formers

    Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Ali- aksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B Lindell, and Sergey Tulyakov. Ac3d: Analyzing and improving 3d camera control in video diffusion trans- formers. InCVPR, 2025. 1, 2

    2025

  2. [2]

    Positional encoding field.arXiv preprint arXiv:2510.20385, 2025

    Yunpeng Bai, Haoxiang Li, and Qixing Huang. Positional encoding field.arXiv preprint arXiv:2510.20385, 2025. 3

    arXiv 2025

  3. [3]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 1, 2, 3

    2021

  4. [4]

    pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction

    David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. InCVPR, 2024. 13

    2024

  5. [5]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 8

    2017

  6. [6]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. InCVPR, 2023. 5, 6, 9, 13, 14, 16

    2023

  7. [7]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. InNAACL-HLT, 2019. 2

    2019

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021. 1, 2

    2021

  9. [9]

    Stable virtual camera: Generative view synthe- sis with diffusion models.arXiv preprint arXiv:2503.14489,

    Hang Gao, Vikram V oleti, Aaryaman Vasishta, Chun-Han Yao, Mark Boss, Philip Torr, Christian Rupprecht, and Varun Jampani. Stable virtual camera: Generative view synthe- sis with diffusion models.arXiv preprint arXiv:2503.14489,

    Pith/arXiv arXiv

  10. [10]

    Cat3d: create anything in 3d with multi-view diffusion models

    Ruiqi Gao, Aleksander Hoły ´nski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: create anything in 3d with multi-view diffusion models. InNeurIPS, 2024. 1, 2

    2024

  11. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 2

    Pith/arXiv arXiv 2025

  12. [12]

    Rotary position embedding for vision transformer

    Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In ECCV, 2024. 2, 3, 13

    2024

  13. [13]

    3d concept learn- ing and reasoning from multi-view images

    Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 3d concept learn- ing and reasoning from multi-view images. InCVPR, 2023. 2

    2023

  14. [14]

    Lrm: Large reconstruction model for single image to 3d

    Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. InICLR, 2024. 2

    2024

  15. [15]

    Odin: a single model for 2d and 3d segmentation

    Ayush Jain, Pushkal Katara, Nikolaos Gkanatsios, Adam W Harley, Gabriel Sarch, Kriti Aggarwal, Vishrav Chaudhary, and Katerina Fragkiadaki. Odin: a single model for 2d and 3d segmentation. InCVPR, 2024. 2

    2024

  16. [16]

    Lvsm: A large view synthesis model with minimal 3d inductive bias

    Haian Jin, Hanwen Jiang, Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. Lvsm: A large view synthesis model with minimal 3d inductive bias. InICLR, 2025. 2, 5, 6, 13

    2025

  17. [17]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InICCV, 2023. 2

    2023

  18. [18]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024. 2, 3

    Pith/arXiv arXiv 2024

  19. [19]

    Eschernet: A generative model for scalable view synthesis

    Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xi- aojuan Qi, and Andrew J Davison. Eschernet: A generative model for scalable view synthesis. InCVPR, 2024. 1, 2, 3, 5, 6

    2024

  20. [20]

    Cameras as relative positional encoding

    Ruilong Li, Brent Yi, Junchen Liu, Hang Gao, Yi Ma, and Angjoo Kanazawa. Cameras as relative positional encoding. InNeurIPS, 2025. 1, 2, 3, 6, 8, 9, 13, 15, 16

    2025

  21. [21]

    Learnable fourier features for multi-dimensional spatial po- sitional encoding

    Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable fourier features for multi-dimensional spatial po- sitional encoding. InNeurIPS, 2021. 1

    2021

  22. [22]

    Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision

    Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision. InCVPR, 2024. 6

    2024

  23. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023. 2

    2023

  24. [24]

    Zero-1-to-3: Zero-shot one image to 3d object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- makov, Sergey Zakharov, and Carl V ondrick. Zero-1-to-3: Zero-shot one image to 3d object. InICCV, 2023. 1, 2

    2023

  25. [25]

    Scaling sequence-to- sequence generative neural rendering.arXiv preprint arXiv:2510.04236, 2025

    Shikun Liu, Kam Woh Ng, Wonbong Jang, Jiadong Guo, Junlin Han, Haozhe Liu, Yiannis Douratsos, Juan C P ´erez, Zijian Zhou, Chi Phung, et al. Scaling sequence-to- sequence generative neural rendering.arXiv preprint arXiv:2510.04236, 2025. 2

    Pith/arXiv arXiv 2025

  26. [26]

    Gta: A geometry-aware attention mechanism for multi-view transformers

    Takeru Miyato, Bernhard Jaeger, Max Welling, and Andreas Geiger. Gta: A geometry-aware attention mechanism for multi-view transformers. InICLR, 2024. 1, 2, 3, 6, 8, 15, 16

    2024

  27. [27]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. InICLR, 2022. 2 10

    2022

  28. [28]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, 2021. 1, 2

    2021

  29. [29]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. InJMLR, 2020. 2

    2020

  30. [30]

    Sam 2: Seg- ment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Seg- ment anything in images and videos. InICLR, 2025. 3

    2025

  31. [31]

    Com- mon objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction

    Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Com- mon objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. InICCV, 2021. 5, 6, 8, 9, 13

    2021

  32. [32]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. InCVPR, 2022. 2

    2022

  33. [33]

    Learning the ropes: Better 2d and 3d position encodings with string

    Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Kumar Avinava Dubey, Ayzaan Wahid, et al. Learning the ropes: Better 2d and 3d position encodings with string. InICML, 2025. 2

    2025

  34. [34]

    Self- attention with relative position representations

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self- attention with relative position representations. InNAACL,

  35. [35]

    Dinov3.arXiv preprint arXiv:2508.10104, 2025

    Oriane Sim ´eoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha ¨el Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025. 1, 3

    Pith/arXiv arXiv 2025

  36. [36]

    A benchmark for the eval- uation of rgb-d slam systems

    J ¨urgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the eval- uation of rgb-d slam systems. InIROS, 2012. 6, 8

    2012

  37. [37]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568, 2024. 2

    2024

  38. [38]

    Bolt3d: Generating 3d scenes in seconds

    Stanislaw Szymanowicz, Jason Y Zhang, Pratul Srinivasan, Ruiqi Gao, Arthur Brussee, Aleksander Holynski, Ricardo Martin-Brualla, Jonathan T Barron, and Philipp Henzler. Bolt3d: Generating 3d scenes in seconds. InICCV, 2025. 2

    2025

  39. [39]

    Lgm: Large multi-view gaus- sian model for high-resolution 3d content creation

    Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaus- sian model for high-resolution 3d content creation. InECCV,

  40. [40]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    Pith/arXiv arXiv 2023

  41. [41]

    Ummenhofer, H

    B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox. Demon: Depth and motion net- work for learning monocular stereo. InCVPR, 2017. 6, 8

    2017

  42. [42]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, 2017. 1, 2

    2017

  43. [43]

    Bullettime: Decoupled control of time and camera pose for video generation.arXiv preprint arXiv:2512.05076, 2025

    Yiming Wang, Qihang Zhang, Shengqu Cai, Tong Wu, Jan Ackermann, Zhengfei Kuang, Yang Zheng, Frano Raji ˇc, Siyu Tang, and Gordon Wetzstein. Bullettime: Decoupled control of time and camera pose for video generation.arXiv preprint arXiv:2512.05076, 2025. 2

    arXiv 2025

  44. [44]

    Sun3d: A database of big spaces reconstructed using sfm and object labels

    Jianxiong Xiao, Andrew Owens, and Antonio Torralba. Sun3d: A database of big spaces reconstructed using sfm and object labels. InCVPR, 2013. 6, 8

    2013

  45. [45]

    Unifying flow, stereo and depth estimation

    Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. InTPAMI, 2023. 2, 6, 7, 8, 13

    2023

  46. [46]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. InICLR, 2025. 3

    2025

  47. [47]

    Unified camera positional encoding for controlled video gen- eration.arXiv preprint arXiv:2512.07237, 2025

    Cheng Zhang, Boying Li, Meng Wei, Yan-Pei Cao, Camilo Cruz Gambardella, Dinh Phung, and Jianfei Cai. Unified camera positional encoding for controlled video gen- eration.arXiv preprint arXiv:2512.07237, 2025. 2

    arXiv 2025

  48. [48]

    Cameras as rays: Pose estimation via ray diffusion

    Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion. InICLR, 2024. 13

    2024

  49. [49]

    Gs-lrm: Large recon- struction model for 3d gaussian splatting

    Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-lrm: Large recon- struction model for 3d gaussian splatting. InECCV, 2024. 2

    2024

  50. [50]

    Stereo magnification: Learning view syn- thesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view syn- thesis using multiplane images. InACM SIGGRAPH, 2018. 5, 6, 8, 13

    2018

  51. [51]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness. InICCV, 2025. 2 11 RayRoPE: Projective Ray Positional Encoding for Multi-view Attention Supplementary Material A. Implementing RayRoPE A.1. Details on Applying RayRoPE The discussion in Sec. 4 focuses on apply...

    2025

  52. [52]

    All other training and evaluation configurations are kept exactly the same as in the original paper. C. Runtime Efficiency Although our method necessitates computingNsets of en- codings and group-wise attention (Sec. A.1), we empiri- Figure 8.Comparison on runtime efficiency. RayRoPE main- tains a runtime efficiency highly comparable to PRoPE, with only a...