REVIEW 2 major objections 5 minor 44 references
DefSLAM: Tracking and Mapping of Deforming Scenes from Monocular Sequences
T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read DefSLAM is the first monocular SLAM that tracks and maps deforming scenes in real time.
desk verdict First monocular SLAM for deforming scenes, convincingly engineered; the real-time headline is undercut by the paper's own reported runtimes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a deformable template: a triangular mesh representing the shape-at-rest of the observed surface, with 3D map points embedded in its facets via barycentric coordinates. Tracking works by minimizing reprojection error together with a deformation energy made of a stretching term on edge lengths, a bending term on discrete mean curvature, and a reference regularizer that pulls nodes toward their rest position. Mapping works through the isometric NRSfM formulation, where the warp between keyframes relates metric tensors and Christoffel symbols under isometry and infinitesimal planarity, producing cubic polynomial equations whose solution gives surface normals; shape-from-normals then recovers the up-to-scale surface, and a Sim(3) alignment fixes its scale.
What would settle it
Run DefSLAM on a monocular sequence of a textured sheet undergoing controlled stretching, with ground-truth 3D shape from a motion-capture marker grid or a patterned stretchable fabric. Measure the per-frame 3D RMS error as a function of stretch magnitude; if error remains low well beyond the isometric regime, the isometry assumption is not load-bearing, and if error degrades sharply, the central claim is confirmed to depend on it.
Extended reading notes
Core claim
The central discovery is that real-time monocular SLAM in deforming scenes is possible by combining two existing non-rigid techniques that had previously been used separately. A deformation tracking thread, based on shape-from-template, treats the current shape-at-rest template as a deformable triangular mesh and jointly minimizes reprojection error with stretching, bending, and reference energies to estimate camera pose and scene deformation at frame rate. A deformation mapping thread, based on isometric non-rigid structure-from-motion with infinitesimal planarity, processes batches of perspective keyframes to compute surface normals, reconstruct an up-to-scale surface, and align it to the existing map under a Sim(3) transformation, thereby updating or extending the template. The two threads run in parallel and only optimize the observed part of the template, giving runtime independent of map size in exploratory sequences. The paper argues that this yields the first monocular SLAM capable of producing accurate 3D models of deforming scenes in real time, including in-vivo medical endoscopy.
Load-bearing premise
The load-bearing premise is that the observed surface deforms isometrically, preserving geodesic distances, and is locally planar; if the scene stretches or shears, the mapping equations and tracking energies no longer describe the deformation.
Editorial extensions
If this is right
- Monocular endoscopy and laparoscopy can be used for real-time 3D reconstruction and camera localization in soft-tissue scenes, without a stereo or depth sensor.
- SLAM systems no longer need to discard or freeze deformed parts of a scene; deformation can be represented explicitly and updated continuously.
- Because only the observed local zone of the template is optimized, the approach scales to exploratory trajectories with growing maps.
- In scenes that are nearly rigid or mildly deforming, the same system behaves like a rigid SLAM system, since isometric NRSfM handles rigid and non-rigid cases alike.
- The incremental normal refinement means later keyframes can improve earlier estimates of surface shape, reducing reconstruction error over time.
Reading between the lines
- The same architecture could be extended to non-isometric deformations by replacing the isometry constraint in the mapping thread with a more general elasticity model, at the cost of extra unknowns; the tracking thread's energy terms already resemble a thin-shell model that could absorb mild stretching.
- A testable consequence not reported in the paper is that tracking accuracy should degrade gracefully with increasing stretch, and a threshold experiment on a stretchable fabric would locate where the isometry assumption breaks.
- The scale drift mechanism identified in the paper suggests that loop-like re-observations could be used as a correction signal: revisiting a previously mapped region provides direct observations of scale that could be fed back to the Sim(3) alignment.
- In surgical practice, a system like this could supply the geometry needed for augmented-reality overlays even when tissue moves and the camera is hand-held, but the real bottleneck for clinical use may be texture and illumination rather than deformation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents DefSLAM, a monocular visual SLAM system for deforming scenes. It combines a shape-from-template (SfT) front-end with an isometric non-rigid structure-from-motion (NRSfM) back-end: a deformation tracking thread estimates camera pose and scene deformation at frame rate, while a deformation mapping thread updates the template at keyframe rate. The method is evaluated on a newly introduced mandala kerchief dataset and on the Hamlyn medical endoscopy dataset, with ablations and sensitivity analyses. The headline claim is that DefSLAM is the first monocular SLAM capable of operating in deforming scenes in real time.
Significance. If the real-time claim is substantiated, the contribution is significant: it is the first monocular SLAM system designed for predominantly deforming scenes, with clear relevance to endoscopic navigation and exploratory trajectories. The paper's main strengths are the clean integration of SfT and NRSfM into a SLAM architecture, the explicit treatment of template initialization and update, the inclusion of a new controlled dataset, and the ablation and sensitivity studies for the deformation energy weights. The work is less about proposing new deformation models than about system-level combination, and the paper is transparent about its assumptions and limitations. The reported experiments and the open-source release (indicated in the text) would support reproducibility, provided the mathematical inconsistencies and the real-time evidence are addressed.
major comments (2)
- [VI and VII] The real-time claim made in the abstract and in Section III is not supported by the reported runtime numbers. Section VI states that, on a 10x10 node mesh, deformation tracking takes approximately 50 ms and deformation mapping approximately 400 ms on an i7-4700HQ. The Mandala dataset is acquired at 30 fps (Section VII-A), so the frame period is 33 ms and, with a keyframe inserted every 10 frames, the mapping thread has a 333 ms budget per keyframe. The reported per-thread times of 50 ms and 400 ms thus exceed the acquisition intervals, even before considering that Section VII states all experiments were run "sequentialized in single-thread." Therefore the central claim of real-time operation is not demonstrated by the reported configuration. Please either provide measured end-to-end throughput in a parallel execution that meets the frame and keyframe rates, or qualify the real-time claim to the conditions under which it actually holds.
- [Eq. (8) and Appendix A(c)] There is an internal inconsistency between the reference regularizer as defined in Eq. (8) and its derivative as reported in Appendix A(c). Equation (8) defines the regularizer as a sum of Euclidean norms, ||V_t_n - V_k_n||, but the derivative in Eq. (27) is given as the constant 1. The derivative of the Euclidean norm of a vector is the unit vector in the direction of the difference, not the constant 1; if the intended error term is the vector V_t_n - V_k_n used in a least-squares framework, the printed objective in Eq. (8) should be a squared norm, not a norm. Since this regularizer is part of the Levenberg-Marquardt optimization in Section IV-C, the Jacobian must be consistent with the objective for the implementation to work as described. Please correct the equation or the derivative, and indicate exactly which form is used in the released code.
minor comments (5)
- [V-F] The initialization description states that the template is "a plane parallel to perpendicular to the camera optical axis"; this is contradictory and should be rephrased to state clearly that the initial surface is assumed to be a plane perpendicular to the optical axis.
- [VI] The sentence "The code will is available as a public git repository" contains a grammatical error; it should be "The code is available as a public git repository."
- [VII-A3] In the sensitivity analysis, the error range is printed as "between 20 and 40 m" but the metric is 3D RMS error in millimeters; the unit should be corrected.
- [VII-A1] Only median values of five executions are reported, without any measure of spread such as interquartile ranges. Given the randomness in ORB detection and the optimization, reporting an error bar would strengthen the quantitative claims.
- [VII-B] The comparison with MISSLAM reports mean accuracy of 3 and 4 mm versus 0.28 and 0.35 mm, but this is not a head-to-head comparison since MISSLAM is stereo and GPU-based. The paper acknowledges this, but the discrepancy should be discussed more explicitly so that readers do not over-interpret the numbers.
Circularity Check
No circularity: DefSLAM integrates previously published, externally validated SfT and NRSfM components; the central claim is not equivalent to its inputs.
full rationale
DefSLAM's derivation chain is a system integration, not a self-referential derivation. The tracking energy (eqs. 3-8) and the NRSfM polynomial system (eqs. 11-14) are explicitly imported from the authors' earlier peer-reviewed works [24] and [33], with the paper stating "We build on the isometric NRSfM proposed in [33]" and "We refer to [33] for further details." Importing a published method is not circular: the paper does not claim to derive those equations from the target result, and the cited methods are independently published and externally evaluated. The only free parameters are the regularizer weights λs=16000, λb=300, λr=0.02, which are tuned and then submitted to a sensitivity analysis; the reported errors are measured against external ground truth (stereo/CT) after per-frame scale alignment, a standard evaluation convention rather than a fitted prediction. The bootstrap between tracking (which uses the current template) and mapping (which refines the template from keyframes) is a feedback loop, but at each step one component is fixed, and the initial template is a plane, so the final map is not equivalent to the input by construction. The runtime inconsistency (50 ms tracking vs. 30 fps input, and single-threaded experiments) is a performance/correctness issue, not a circularity issue. Self-citations [24] and [33] are present but are not load-bearing circularities because they are independent support. No circular step can be exhibited, so the score is 0.
Assumptions & free parameters
free parameters (4)
- λs (stretching weight) =
16000
- λb (bending weight) =
300
- λr (reference regularizer weight) =
0.02
- keyframe insertion interval =
10 frames
assumptions (5)
- domain assumption Scene deformation is isometric (geodesic distances between surface points are preserved).
- domain assumption Surface is infinitesimally planar (locally a plane) in NRSfM.
- standard math Camera is calibrated and follows a pinhole model.
- ad hoc to paper Initial template is a plane perpendicular to the camera optical axis.
- domain assumption Inter-frame deformation is small enough for frame-to-frame active matching with ORB descriptors.
Cite this review
Pith. "Pith review of DefSLAM: Tracking and Mapping of Deforming Scenes from Monocular Sequences." pith.science (2026). https://pith.science/paper/33XKVCOE
@misc{pith2026190808918,
author = {Pith},
title = {Pith review of: DefSLAM: Tracking and Mapping of Deforming Scenes from Monocular Sequences},
year = {2026},
howpublished = {\url{https://pith.science/paper/33XKVCOE}},
note = {Machine review of arXiv:1908.08918}
}
read the original abstract
Monocular SLAM algorithms perform robustly when observing rigid scenes, however, they fail when the observed scene deforms, for example, in medical endoscopy applications. We present DefSLAM, the first monocular SLAM capable of operating in deforming scenes in real-time. Our approach intertwines Shape-from-Template (SfT) and Non-Rigid Structure-from-Motion (NRSfM) techniques to deal with the exploratory sequences typical of SLAM. A deformation tracking thread recovers the pose of the camera and the deformation of the observed map, at frame rate, by means of SfT processing a template that models the scene shape-at-rest. A deformation mapping thread runs in parallel with the tracking to update the template, at keyframe rate, by means of an isometric NRSfM processing a batch of full perspective keyframes. In our experiments, DefSLAM processes close-up sequences of deforming scenes, both in a laboratory controlled experiment and in medical endoscopy sequences, producing accurate 3D models of the scene with respect to the moving camera.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
- [1]
- [2]
-
[3]
A. Agudo and F. Moreno-Noguer. Simultaneous pose and non-rigid shape with particle dynamics. In CVPR, 2015
work page 2015
- [4]
-
[5]
A. Bartoli, Y . G ´erard, F. Chadebecq, T. Collins, and D. Pizarro. Shape- from-template. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(10):2099–2118, 2015
work page 2015
-
[6]
G. Bradski. The OpenCV Library. Dr. Dobb’s Journal of Software Tools, 2000
2000
-
[7]
Bregler, A
C. Bregler, A. Hertzmann, and H. Biermann. Recovering non-rigid 3D shape from image streams. In CVPR, 2000
2000
-
[8]
A. Chhatkuli, D. Pizarro, and A. Bartoli. Non-rigid shape-from-motion for isometric surfaces using infinitesimal planarity. In BMVC, 2014
work page 2014
Show all 44 references
-
[9]
Chhatkuli, D
A. Chhatkuli, D. Pizarro, A. Bartoli, and T. Collins. A stable analytical framework for isometric shape-from-template by surface integration. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(5):833–850, 2017
2017
-
[10]
Chhatkuli, D
A. Chhatkuli, D. Pizarro, T. Collins, and A. Bartoli. Inextensible non- rigid shape-from-motion by second-order cone programming. In CVPR, 2016
2016
-
[11]
Collins and A
T. Collins and A. Bartoli. Locally affine and planar deformable surface reconstruction from video. In International Workshop on Vision, Modeling and Visualization , 2010
2010
-
[12]
Y . Dai, H. Li, and M. He. A simple prior-free method for non-rigid structure-from-motion factorization. International Journal of Computer Vision, 107(2):101–122, 2014
2014
-
[13]
A. J. Davison. Real-time simultaneous localisation and mapping with a single camera. In null, page 1403. IEEE, 2003
2003
-
[14]
Engel, V
J. Engel, V . Koltun, and D. Cremers. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(3):611– 625, 2017
2017
-
[15]
M. S. Floater. Mean value coordinates. Computer Aided Geometric Design, 20(1):19–27, 2003
2003
-
[16]
Gao and R
W. Gao and R. Tedrake. Surfelwarp: Efficient non-volumetric single view dynamic reconstruction. In Robotics: Science and System (RSS) , 2018
2018
-
[17]
R. Garg, A. Roussos, and L. Agapito. Dense variational reconstruction of non-rigid surfaces from monocular video. In CVPR, 2013
2013
-
[18]
P. F. Gotardo and A. M. Martinez. Kernel non-rigid structure from motion. In ICCV, 2011
2011
-
[19]
P. F. Gotardo and A. M. Martinez. Non-rigid structure from motion with complementary rank-3 spaces. In CVPR, 2011
2011
-
[20]
O. G. Grasa, E. Bernal, S. Casado, I. Gil, and J. Montiel. Visual slam for handheld monocular endoscope. IEEE Transactions on Medical Imaging, 33(1):135–146, 2014
2014
-
[21]
Innmann, M
M. Innmann, M. Zollh ¨ofer, M. Nießner, C. Theobalt, and M. Stam- minger. V olumedeform: Real-time volumetric non-rigid reconstruction. In ECCV, 2016
2016
-
[22]
Klein and D
G. Klein and D. Murray. Parallel tracking and mapping for small AR workspaces. In ISMAR, 2007
2007
-
[23]
K ¨ummerle, G
R. K ¨ummerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard. g2o: A general framework for graph optimization. In ICRA. IEEE, 2011
2011
-
[24]
Lamarca and J
J. Lamarca and J. M. M. Montiel. Camera tracking for SLAM in deformable maps. In 4th Inter. Workshop on Recovering 6D Object Pose. In ECCVw, 2018
2018
-
[25]
Mahmoud, T
N. Mahmoud, T. Collins, A. Hostettler, L. Soler, C. Doignon, and J. M. M. Montiel. Live tracking and dense reconstruction for hand- held monocular endoscopy. IEEE Transactions on Medical Imaging , 2018
2018
-
[26]
Marmol, A
A. Marmol, A. Banach, and T. Peynot. Dense-arthroslam: Dense intra- articular 3-d reconstruction with robust localization prior for arthroscopy. IEEE Robotics and Automation Letters , 4(2):918–925, 2019
2019
-
[27]
Moreno-Noguer and J
F. Moreno-Noguer and J. M. Porta. Probabilistic simultaneous pose and non-rigid shape recovery. In CVPR, 2011
2011
-
[28]
Mountney, D
P. Mountney, D. Stoyanov, and G.-Z. Yang. Three-dimensional tissue deformation recovery and tracking. IEEE Signal Processing Magazine , 27(4):14–24, 2010
2010
-
[29]
Mur-Artal, J
R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos. ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Transactions on Robotics, 31(5):1147–1163, 2015
2015
-
[30]
R. A. Newcombe, D. Fox, and S. M. Seitz. Dynamicfusion: Reconstruc- tion and tracking of non-rigid scenes in real-time. In CVPR, 2015
2015
-
[31]
D. T. Ngo, J. ¨Ostlund, and P. Fua. Template-based monocular 3D shape recovery using laplacian meshes. IEEE Transactions on Pattern Analysis and Machine Intelligence , 38(1):172–187, 2016
2016
-
[32]
Paladini, A
M. Paladini, A. Del Bue, M. Stosic, M. Dodig, J. Xavier, and L. Agapito. Factorization for non-rigid and articulated structure using metric projec- tions. In CVPR, 2009
2009
-
[33]
Parashar, D
S. Parashar, D. Pizarro, and A. Bartoli. Isometric non-rigid shape-from- motion with riemannian geometry solved in linear time. IEEE Transac- tions on Pattern Analysis and Machine Intelligence , 40(10):2442–2454, 2017
2017
-
[34]
Pizarro, R
D. Pizarro, R. Khan, and A. Bartoli. Schwarps: Locally projective image warps based on 2d schwarzian derivatives. International Journal of Computer Vision, 119(2):93–109, 2016
2016
-
[35]
Salzmann and P
M. Salzmann and P. Fua. Linear local models for monocular reconstruc- tion of deformable surfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):931–944, 2011
2011
-
[36]
J. Song, J. Wang, L. Zhao, S. Huang, and G. Dissanayake. Dynamic reconstruction of deformable soft-tissue with stereo scope in minimal invasive surgery. IEEE Robotics and Automation Letters , 3(1):155–162, 2017
2017
-
[37]
J. Song, J. Wang, L. Zhao, S. Huang, and G. Dissanayake. Mis- slam: Real-time large-scale dense deformable slam system in minimal invasive surgery based on heterogeneous computing. IEEE Robotics and Automation Letters, 3(4):4068–4075, 2018
2018
-
[38]
Sorkine and M
O. Sorkine and M. Alexa. As-rigid-as-possible surface modeling. In Eurographics, 2007
2007
-
[39]
Stoyanov, G
D. Stoyanov, G. P. Mylonas, F. Deligianni, A. Darzi, and G. Z. Yang. Soft-tissue motion tracking and structure estimation for robotic assisted mis procedures. In MICAI, 2005
2005
-
[40]
Stoyanov, M
D. Stoyanov, M. V . Scarzanella, P. Pratt, and G.-Z. Yang. Real- time stereo reconstruction in robotically assisted minimally invasive surgery. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 275–282. Springer, 2010
2010
-
[41]
R. W. Sumner, J. Schmid, and M. Pauly. Embedded deformation for shape manipulation. In ACM SIGGRAPH 2007 papers , pages 80–es. 2007
2007
-
[42]
Taylor, A
J. Taylor, A. D. Jepson, and K. N. Kutulakos. Non-rigid structure from locally-rigid motion. In CVPR, 2010
2010
-
[43]
Vicente and L
S. Vicente and L. Agapito. Soft inextensibility constraints for template- free non-rigid reconstruction. In ECCV, 2012. APPENDIX A. Derivatives of Regularizers We show the Jacobian terms of the regularizers to prove that it does not have singularities: a) Stretching: The strec...
2012
-
[2016]
Since 2020 he is coordinating the EU FET EndoMapper grant aimed to bring visual SLAM to medical intracorporeal scenes
2020
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.