BA-T: An Iterative Transformer for Two-View Bundle Adjustment
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-28 10:23 UTCgrok-4.3pith:UXSID53Trecord.jsonopen to challenge →
The pith
BA-T replaces deep attention stacks with a single repeatable lightweight layer that performs bundle-adjustment style updates for two-view 3D reconstruction.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
BA-T implements bundle adjustment as an iterative information propagation process between poses and local geometry realized as a single lightweight repeatable layer in implicit token space. This layer refines predictions from latent residuals rather than relying on deep cross-view attention stacks, producing progressive improvements in pose and reconstruction accuracy together with stronger cross-view consistency.
What carries the argument
The BA-T layer: a single lightweight transformer layer that executes BA-style structured updates as a repeatable operation in token space.
If this is right
- Pose and point accuracy increase with each additional iteration of the BA-T layer.
- Cross-view consistency exceeds that obtained from conventional deep decoder stacks.
- Performance matches or exceeds substantially larger models while using 16 percent of their decoder parameters.
- The architecture supplies a compact structural alternative to depth-heavy attention for accurate 3D reconstruction.
Where Pith is reading between the lines
- The same lightweight update layer could be stacked or adapted for three or more input views without redesigning the core mechanism.
- Training might converge faster if the BA-style residual update is initialized from classical bundle-adjustment solutions on the same data.
- Runtime cost in real-time pipelines could drop further if the number of iterations is made input-dependent rather than fixed.
Load-bearing premise
One lightweight layer can faithfully carry out the structured geometric updates of bundle adjustment inside implicit token representations.
What would settle it
Measure whether reconstruction error and cross-view consistency continue to improve after multiple iterations on a held-out two-view benchmark or plateau at the level of a single pass.
Figures
read the original abstract
Feed-forward models for 3D reconstruction have achieved strong performance using deep cross-view attention to exchange information across images. However, these approaches often depend on heavy decoder stacks and lack a structured mechanism for geometry refinement, resulting in poor multi-view consistency. We address this by drawing inspiration from classical bundle adjustment (BA), which can be viewed as an iterative information propagation process between poses and local geometry. Inspired by BA, we propose BA-T, an iterative Transformer that implements BA-style structured updates as a repeatable layer in implicit token space. Instead of relying on deep attention stacks, BA-T refines predictions based on latent residual by a single lightweight layer. Experiments demonstrate that BA-T progressively improves pose and reconstruction accuracy across iterations, achieves stronger cross-view consistency than conventional decoders, and matches or surpasses substantially larger models while using only 16% of their decoder parameters. BA-T provides a compact, efficient, and structural alternative to depth-heavy attention, enabling accurate 3D reconstruction within a lightweight architecture. The code will be made publicly at https://github.com/zhangganlin/BA-T.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes BA-T, an iterative Transformer for two-view bundle adjustment that draws from classical BA as an information-propagation process between poses and local geometry. It replaces heavy decoder stacks with a single repeatable lightweight layer that performs BA-style structured updates in implicit token space by refining predictions from latent residuals. The central claims are that this yields progressive gains in pose/reconstruction accuracy across iterations, stronger cross-view consistency than conventional decoders, and performance matching or exceeding much larger models while using only 16% of their decoder parameters.
Significance. If the claimed structural equivalence to BA holds and is shown to be non-circular, the result would supply a compact, parameter-efficient architectural primitive for multi-view geometry that could replace depth-heavy attention in feed-forward 3D reconstruction pipelines. The explicit promise of public code strengthens reproducibility.
major comments (2)
- [Abstract / §3 (method)] Abstract and method description: the claim that the repeatable lightweight layer 'implements BA-style structured updates' and refines 'based on latent residual' is load-bearing for all consistency and efficiency assertions, yet no equations are supplied that map the layer operations (attention, residual, or token interactions) onto classical BA quantities such as the normal equations, Schur complement, or explicit pose-point information propagation. Without this mapping it remains possible that observed gains arise from iteration count or residual connections alone.
- [Abstract / §4 (experiments)] Experimental section: the abstract asserts progressive accuracy improvement, stronger consistency, and parameter-efficient superiority, but the provided text supplies no dataset names, baseline architectures, error metrics (e.g., rotation/translation error, reprojection), ablation controls on layer depth versus iteration count, or statistical significance tests. These details are required to substantiate the cross-model comparison at 16% decoder parameters.
minor comments (2)
- [Abstract] The abstract states that code will be released at a GitHub URL; confirming the repository contains the exact training and evaluation scripts used for the reported numbers would aid verification.
- [§3] Notation for pose and point tokens should be introduced once with explicit dimensionality before the layer description to avoid ambiguity in the implicit token space.
Simulated Author's Rebuttal
We thank the referee for the constructive comments. We address each major comment below and indicate the planned revisions.
read point-by-point responses
-
Referee: [Abstract / §3 (method)] Abstract and method description: the claim that the repeatable lightweight layer 'implements BA-style structured updates' and refines 'based on latent residual' is load-bearing for all consistency and efficiency assertions, yet no equations are supplied that map the layer operations (attention, residual, or token interactions) onto classical BA quantities such as the normal equations, Schur complement, or explicit pose-point information propagation. Without this mapping it remains possible that observed gains arise from iteration count or residual connections alone.
Authors: We agree that the absence of an explicit mapping leaves the structural claim open to the interpretation raised. The layer is motivated by viewing BA as iterative information propagation between poses and points, realized via attention and residuals in token space, but the manuscript does not derive or equate the operations to the normal equations or Schur complement. In revision we will add a concise subsection in §3 that supplies a conceptual correspondence (e.g., how cross-view attention approximates pose-point message passing and how the residual step parallels the BA update), while acknowledging it is an implicit rather than algebraic equivalence. This will clarify the intended source of the observed gains. revision: yes
-
Referee: [Abstract / §4 (experiments)] Experimental section: the abstract asserts progressive accuracy improvement, stronger consistency, and parameter-efficient superiority, but the provided text supplies no dataset names, baseline architectures, error metrics (e.g., rotation/translation error, reprojection), ablation controls on layer depth versus iteration count, or statistical significance tests. These details are required to substantiate the cross-model comparison at 16% decoder parameters.
Authors: We accept that the experimental reporting must be expanded for the claims to be fully substantiated. The current manuscript text does not enumerate the required specifics. In the revised version we will augment §4 with explicit dataset names, baseline architectures, the precise error metrics, ablations that isolate iteration count from layer depth, and any statistical tests performed, thereby supporting the progressive improvement and 16 % parameter-efficiency statements. revision: yes
Circularity Check
No circularity: architectural claim stands independent of inputs
full rationale
The paper presents BA-T as an iterative transformer layer inspired by classical bundle adjustment for structured updates in token space. No equations, fitted parameters, or self-citations are shown that reduce the claimed consistency gains or parameter efficiency to a definitional equivalence or statistical forcing. The derivation chain consists of an inspiration step followed by experimental validation; the layer is not shown to be equivalent to its inputs by construction, nor does any load-bearing premise collapse to a prior self-citation. This is the common case of a self-contained architectural proposal.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
In: ECCV
Agarwal, S., Snavely, N., Seitz, S.M., Szeliski, R.: Bundle adjustment in the large. In: ECCV . pp. 29–42. Springer (2010)
2010
-
[2]
In: ACCV
Alismail, H., Browning, B., Lucey, S.: Photometric bundle adjustment for vision-based slam. In: ACCV . pp. 324–341. Springer (2016)
2016
-
[3]
In: ECCV
Avetisyan, A., Xie, C., Howard-Jenkins, H., Yang, T.Y ., Aroudj, S., Patra, S., Zhang, F., Frost, D., Holland, L., Orme, C., et al.: Scenescript: Reconstructing scenes with an autoregressive structured language model. In: ECCV . pp. 247–263. Springer (2024)
2024
-
[4]
In: NeurIPS (2021)
Baruch, G., Chen, Z., Dehghan, A., Dimry, T., Feigin, Y ., Fu, P., Gebauer, T., Joffe, B., Kurz, D., Schwartz, A., Shulman, E.: ARKitScenes - a diverse real-world dataset for 3D indoor scene understanding using mobile RGB-D data. In: NeurIPS (2021)
2021
-
[5]
In: CVPR
Cabon, Y ., Stoffl, L., Antsfeld, L., Csurka, G., Chidlovskii, B., Revaud, J., Leroy, V .: MUSt3R: Multi-view network for stereo 3D reconstruction. In: CVPR. pp. 1050–1060 (2025)
2025
-
[6]
(eds.): SLAM Handbook
Carlone, L., Kim, A., Barfoot, T., Cremers, D., Dellaert, F. (eds.): SLAM Handbook. From Localization and Mapping to Spatial Intelligence. Cambridge University Press (2026)
2026
-
[7]
In: CVPR (2017)
Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: ScanNet: Richly- annotated 3D reconstructions of indoor scenes. In: CVPR (2017)
2017
-
[8]
ACM TOG (2017)
Dai, A., Nießner, M., Zollöfer, M., Izadi, S., Theobalt, C.: BundleFusion: Real-time globally consistent 3D reconstruction using on-the-fly surface re-integration. ACM TOG (2017)
2017
-
[9]
IEEE TPAMI29(6), 1052–1067 (2007)
Davison, A.J., Reid, I.D., Molton, N.D., Stasse, O.: MonoSLAM: Real-time single camera SLAM. IEEE TPAMI29(6), 1052–1067 (2007)
2007
-
[10]
In: CVPR
Dong, S., Wang, S., Liu, S., Cai, L., Fan, Q., Kannala, J., Yang, Y .: Reloc3r: Large-scale training of relative camera pose regression for generalizable, fast, and accurate visual localization. In: CVPR. pp. 16739–16752 (2025)
2025
-
[11]
In: ICLR (2021)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
2021
-
[12]
IEEE TPAMI40(3), 611–625 (2017)
Engel, J., Koltun, V ., Cremers, D.: Direct sparse odometry. IEEE TPAMI40(3), 611–625 (2017)
2017
-
[13]
In: ECCV
Engel, J., Schöps, T., Cremers, D.: LSD-SLAM: Large-scale direct monocular SLAM. In: ECCV . pp. 834–849. Springer (2014)
2014
-
[14]
In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Gao, X., Wang, R., Demmel, N., Cremers, D.: LDSO: Direct sparse odometry with loop closure. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 2198–2204. IEEE (2018)
2018
-
[15]
In: ICCV
Hagemann, A., Knorr, M., Stiller, C.: Deep geometry-aware camera self-calibration from video. In: ICCV . pp. 3438–3448 (October 2023)
2023
-
[16]
Cambridge university press (2003)
Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)
2003
-
[17]
In: International Conference on 3D Vision (3DV)
Keetha, N., Müller, N., Schönberger, J., Porzi, L., Zhang, Y ., Fischer, T., Knapitsch, A., Zauss, D., Weber, E., Antunes, N., Luiten, J., Lopez-Antequera, M., Bulò, S.R., Richardt, C., Ramanan, D., Scherer, S., Kontschieder, P.: MapAnything: Universal feed-forward metric 3D reconstruction. In: International Conference on 3D Vision (3DV). IEEE (2026)
2026
-
[18]
In: ECCV
Leroy, V ., Cabon, Y ., Revaud, J.: Grounding image matching in 3D with MASt3R. In: ECCV . pp. 71–91. Springer (2024)
2024
-
[19]
Depth Anything 3: Recovering the Visual Space from Any Views
Lin, H., Chen, S., Liew, J.H., Chen, D.Y ., Li, Z., Shi, G., Feng, J., Kang, B.: Depth Anything 3: Recovering the visual space from any views. arXiv preprint arXiv:2511.10647 (2025)
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[20]
In: ECCV
Liu, S., Gao, Y ., Zhang, T., Pautrat, R., Schönberger, J.L., Larsson, V ., Pollefeys, M.: Robust incremental structure-from-motion with hybrid features. In: ECCV . pp. 249–269. Springer (2024)
2024
-
[21]
Decoupled Weight Decay Regularization
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017) 10
work page internal anchor Pith review Pith/arXiv arXiv 2017
-
[22]
IEEE transactions on robotics31(5), 1147–1163 (2015)
Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE transactions on robotics31(5), 1147–1163 (2015)
2015
-
[23]
Springer (2006)
Nocedal, J., Wright, S.J.: Numerical optimization. Springer (2006)
2006
-
[24]
In: ECCV
Pan, L., Baráth, D., Pollefeys, M., Schönberger, J.L.: Global structure-from-motion revisited. In: ECCV . pp. 58–77. Springer (2024)
2024
-
[25]
In: ICCV
Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: ICCV . pp. 4195–4205 (2023)
2023
-
[26]
Flow4r: Unifying 4d reconstruction and tracking with scene flow
Qian, S., Zhang, G., Wu, S., Cremers, D.: Flow4R: Unifying 4d reconstruction and tracking with scene flow. arXiv preprint arXiv:2602.14021 (2026)
-
[27]
In: ICCV (2021)
Reizenstein, J., Shapovalov, R., Henzler, P., Sbordone, L., Labatut, P., Novotny, D.: Common objects in 3D: Large-scale learning and evaluation of real-life 3D category reconstruction. In: ICCV (2021)
2021
-
[28]
In: CVPRW (2025)
Sandström, E., Zhang, G., Tateno, K., Oechsle, M., Niemeyer, M., Zhang, Y ., Patel, M., Van Gool, L., Oswald, M., Tombari, F.: Splat-SLAM: Globally optimized rgb-only SLAM with 3D gaussians. In: CVPRW (2025)
2025
-
[29]
In: CVPR
Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: CVPR. pp. 4104–4113 (2016)
2016
-
[30]
In: CVPR
Shotton, J., Glocker, B., Zach, C., Izadi, S., Criminisi, A., Fitzgibbon, A.: Scene coordinate regression forests for camera relocalization in RGB-D images. In: CVPR. pp. 2930–2937 (2013)
2013
-
[31]
The Replica Dataset: A Digital Replica of Indoor Spaces
Straub, J., Whelan, T., Ma, L., Chen, Y ., Wijmans, E., Green, S., Engel, J.J., Mur-Artal, R., Ren, C., Verma, S., Clarkson, A., Yan, M., Budge, B., Yan, Y ., Pan, X., Yon, J., Zou, Y ., Leon, K., Carter, N., Briales, J., Gillingham, T., Mueggler, E., Pesqueira, L., Savva, M., Batra, D., Strasdat, H.M., Nardi, R.D., Goesele, M., Lovegrove, S., Newcombe, R...
work page internal anchor Pith review Pith/arXiv arXiv 1906
-
[32]
In: IROS (Oct 2012)
Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of RGB-D SLAM systems. In: IROS (Oct 2012)
2012
-
[33]
In: ICLR (2019)
Tang, C., Tan, P.: BA-Net: Dense bundle adjustment network. In: ICLR (2019)
2019
-
[34]
In: ECCV
Teed, Z., Deng, J.: RAFT: Recurrent all-pairs field transforms for optical flow. In: ECCV . pp. 402–419. Springer (2020)
2020
-
[35]
NeurIPS34, 16558–16569 (2021)
Teed, Z., Deng, J.: DROID-SLAM: Deep visual SLAM for monocular, stereo, and RGB-D cameras. NeurIPS34, 16558–16569 (2021)
2021
-
[36]
NeurIPS36, 39033–39051 (2023)
Teed, Z., Lipson, L., Deng, J.: Deep patch visual odometry. NeurIPS36, 39033–39051 (2023)
2023
-
[37]
In: International workshop on vision algorithms
Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment—a modern synthesis. In: International workshop on vision algorithms. pp. 298–372. Springer (1999)
1999
-
[38]
In: 3DV (2025)
Wang, H., Agapito, L.: 3D reconstruction with spatial memory. In: 3DV (2025)
2025
-
[39]
In: CVPR (2025)
Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotny, D.: VGGT: Visual geometry grounded transformer. In: CVPR (2025)
2025
-
[40]
In: CVPR
Wang, J., Karaev, N., Rupprecht, C., Novotny, D.: VGGSfM: Visual geometry grounded deep structure from motion. In: CVPR. pp. 21686–21697 (2024)
2024
-
[41]
In: CVPR
Wang, Q., Zhang, Y ., Holynski, A., Efros, A.A., Kanazawa, A.: Continuous 3D perception model with persistent state. In: CVPR. pp. 10510–10522 (2025)
2025
-
[42]
In: CVPR
Wang, S., Leroy, V ., Cabon, Y ., Chidlovskii, B., Revaud, J.: DUSt3R: Geometric 3D vision made easy. In: CVPR. pp. 20697–20709 (2024)
2024
-
[43]
Wang, Y ., Zhou, J., Zhu, H., Chang, W., Zhou, Y ., Li, Z., Chen, J., Pang, J., Shen, C., He, T.: π3: Scalable permutation-equivariant visual geometry learning (2025)
2025
-
[44]
Resplat: Learning recurrent gaussian splats.arXiv preprint arXiv:2510.08575, 2025
Xu, H., Barath, D., Geiger, A., Pollefeys, M.: Resplat: Learning recurrent gaussian splats. arXiv preprint arXiv:2510.08575 (2025)
-
[45]
In: CVPR
Yang, N., Stumberg, L.v., Wang, R., Cremers, D.: D3VO: Deep depth, deep pose and deep uncertainty for monocular visual odometry. In: CVPR. pp. 1281–1292 (2020)
2020
-
[46]
In: ECCV
Yang, N., Wang, R., Stuckler, J., Cremers, D.: Deep virtual stereo odometry: Leveraging deep depth prediction for monocular direct sparse odometry. In: ECCV . pp. 817–833 (2018) 11
2018
-
[47]
In: ICCV (2023)
Yeshwanth, C., Liu, Y .C., Nießner, M., Dai, A.: ScanNet++: A high-fidelity dataset of 3D indoor scenes. In: ICCV (2023)
2023
-
[48]
arXiv preprint arXiv:2509.16909 (2025)
Yuan, Y ., Chen, Z., Li, K., Wang, W., Zhao, H.: SLAM-Former: Putting SLAM into one transformer. arXiv preprint arXiv:2509.16909 (2025)
-
[49]
arXiv preprint arXiv:2509.01584 (2025)
Zhang, G., Qian, S., Wang, X., Cremers, D.: ViSTA-SLAM: Visual SLAM with symmetric two-view association. arXiv preprint arXiv:2509.01584 (2025)
-
[50]
arXiv preprint arXiv:2403.19549 (2024)
Zhang, G., Sandström, E., Zhang, Y ., Patel, M., Van Gool, L., Oswald, M.R.: GlORIE- SLAM: Globally optimized RGB-only implicit encoding point cloud SLAM. arXiv preprint arXiv:2403.19549 (2024)
-
[51]
arXiv preprint arXiv:2411.17982 (2024)
Zhang, W., Cheng, Q., Skuddis, D., Zeller, N., Cremers, D., Haala, N.: HI-SLAM2: Geometry- aware gaussian SLAM for fast monocular scene reconstruction. arXiv preprint arXiv:2411.17982 (2024)
-
[52]
Zhang, W., Sun, T., Wang, S., Cheng, Q., Haala, N.: HI-SLAM: Monocular real-time dense mapping with hybrid implicit fields. IEEE Robotics and Automation Letters9(2), 1548–1555 (2023) 12 BA-T: An Iterative Transformer for Two-View Bundle Adjustment Appendix A Architecture and Training Details Architecture Details.In BA-T, tokens have dimension D= 768 . In ...
2023
-
[53]
BA-Net [33], RAFT [34], ReSplat [44], and BA-T all share an iterative refinement flavor, but the differences are substantial
[39] [19] [17] [18] [42] [49] (k=1/2/3/4) Running time↓(ms) 160.93 128.71 78.14 41.13 93.25 66.47 37.96 21.40 / 24.65 / 28.32 /30.92 Decoder size↓(M) / 605 765 171 227 227 11338 Peak GPU mem↓(GB) 2.43 4.93 6.52 3.23 2.72 2.02 1.761.32 E Discussion on Iteration and Refinement Behavior E.1 Comparison with Unrolling and Recurrent Methods Although BA-T is ins...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.