Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo Images

Alexander Schlaefer; Christian Betz; Konrad Reuter; Lennart Maack; Sarah Latus; Suresh Guttikonda; Tobias Maurer

REVIEW 4 major objections 6 minor 6 references

Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo Images

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that two CoTracker3 models, one tracking points over time and one matching points across a stereo pair, can estimate 3D tissue motion from endoscopic images with mean errors near 1.1 mm on chicken tissue.

desk verdict A credible feasibility study of using CoTracker as a stereo matcher for surgical tissue tracking, but the headline 1.1 mm figure may lean on an affine calibration that masks stereo bias, and the evidence is thin. read the letter →

arxiv 2508.07851 v1 pith:4P6JUTNW submitted 2025-08-11 cs.CV

Konrad Reuter , Suresh Guttikonda , Sarah Latus , Lennart Maack , Christian Betz , Tobias Maurer , Alexander Schlaefer This is my paper · ORCID

classification cs.CV

keywords TrackingAnyPointCoTracker3stereovisionmarkerlesstissueendoscopicsurgery3Dminimallyinvasive

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that Tracking Any Point (TAP) correspondence networks, originally built to track arbitrary pixels through video, can be repurposed as stereo matchers and thereby produce markerless 3D tissue tracking in endoscopic surgery. The proposed pipeline uses one CoTracker3 model to follow a grid of query points over time in the left camera, and a second CoTracker3 model to find those points in the right camera of each stereo pair, turning their disparity into depth and lateral position. On a clinical laparoscopic setup with a robot arm providing ground truth, the method reports mean Euclidean errors around 1.1 mm at 10 mm/s on chicken tissue, degrading to roughly 3.5 mm at 80 mm/s, at real-time frame rates. A sympathetic reader would care because, if the result transfers to real surgeries, it would make navigation, safety warnings near delicate structures, and context-aware robotic assistance possible without markers or hand-tuned stereo feature matchers. The paper is careful to frame this as feasibility: the errors are still too high for immediate clinical use, and validation on real surgical scenes is still missing.

What carries the argument

Two instances of CoTracker3, a Tracking Any Point network that outputs dense long-range correspondences for arbitrary query points in video. The online instance processes eight-frame batches from one camera for temporal tracking; the offline instance is given each stereo pair as if it were a two-frame video, producing spatial correspondences whose disparity yields depth (Z) and, with the median over a grid of points, lateral position (X, Y).

What would settle it

Take a calibrated stereo pair of a feature-poor surface, compute the offline CoTracker3's disparity, and compare the resulting depth against a known structure or robot ground truth; if the two-frame stereo matches produce substantially larger errors than the temporal eight-frame matches on the same surface, the central assumption that TAP training transfers to stereo is falsified. A second check is to run the full pipeline on real surgical stereo video and compare tracked 3D positions with an independent tracking system; if errors exceed the phantom-reported values at comparable speeds, the cl

Watch

Extended reading notes

Core claim

The central claim is that 2D TAP networks can serve as the complete correspondence engine for 3D tissue tracking in stereo endoscopy. Instead of using a dedicated stereo matching algorithm, the method feeds a rectified left-right pair into an offline CoTracker3 model and treats it as a two-frame tracking problem; the resulting matches give disparity, from which depth and lateral 3D coordinates are computed. A separate online CoTracker3 tracks the same query grid through time in the left view, and the median over the point ensemble stabilizes each axis estimate. Evaluated with a robotic arm moving two phantoms under a clinical laparoscope, the full 3D pipeline achieves mean Euclidean errors o

Load-bearing premise

The load-bearing premise is that CoTracker3, trained for temporal tracking in natural video, will match points reliably across a two-frame stereo pair; the paper itself notes in its discussion that TAP models are not inherently designed for stereo matching, so if spatial matching degrades without motion context, the depth channel of the pipeline fails.

Editorial extensions

If this is right

Markerless 3D tracking in minimally invasive surgery can be built entirely from 2D correspondence networks, without markers, classic stereo features, or a dedicated 3D tracker.
Improvements in TAP models should transfer directly to surgical 3D tracking, since the same model family handles both the temporal and the stereo correspondence task.
At low to moderate tissue velocities, reported errors around 1.1 to 2 mm on chicken tissue are in a range that could support navigation and safety feedback, pending clinical validation.
The pipeline runs in real time (about 33 FPS with a 3x3 grid), so it can be embedded in a live laparoscopic video stream.
Accuracy falls at higher velocities and on feature-poor, reflective surfaces, meaning practical deployment would need speed constraints, texture-aware point selection, or both.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper only evaluates rigid motion imposed by a robot arm; since each query point is tracked independently, the same pipeline could plausibly estimate non-rigid tissue deformation, but that extension is untested here.
The optimal grid size around nine points suggests that adaptive selection of informative features would outperform uniform sampling; the paper identifies point selection as future work but does not explore it experimentally.
Using a shared encoder for the two CoTracker3 instances could reduce the current two-pass processing of each image, which the paper notes as a computational inefficiency.
A direct comparison against a classical stereo matcher or a deep stereo network on the same robot-ground-truth data would isolate how much of the accuracy comes from the TAP repurposing rather than from ensemble averaging; the paper does not include such a baseline.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper does something genuinely new: it takes the CoTracker3 point-tracking model and uses a second instance to match points across a stereo pair, then combines temporal and stereo correspondences to get 3D positions. That repurposing isn't in the literature they cite, and the two-model pipeline is concrete. The experimental setup is also a step up from many medical vision papers: a robot arm moves the target along logged trajectories, and a checkerboard calibration gives an independent ground-truth coordinate frame. The discussion is honest about limitations, including that TAP models aren't designed for two-frame stereo matching.

The main soft spot is the calibration. Section 2.2 says an 'affine transformation' aligns camera and robot coordinates, but doesn't say whether it's rigid (6 DOF), similarity (7 DOF, with scale), or full affine (12 DOF with shear). If it's the latter, it can absorb exactly the kind of systematic stereo error you'd get from a wrong baseline or focal length: the fit moves the reconstructed points to the robot positions, and the 0.124 mm residual on unseen checkerboard positions is so small that it suggests the transform is compensating for more than just coordinate misalignment. That doesn't necessarily break the method—if the correction is consistent across the volume, it's arguably part of the system—but it means the reported 1.1 mm error isn't a measure of raw stereo reconstruction accuracy. The authors should report a rigid-transform calibration as a comparison, or clearly justify why an affine transform is appropriate.

There are also smaller issues. The main result figures plot mean errors only, with no error bars or per-trajectory spread, and only three random trajectories per velocity. The point-density tradeoff (Figure 7) is tuned on the same chicken data used for evaluation, which is mild but worth noting. No code or data are released, and there are no baselines against classical stereo or other learned matchers. These are all fixable and none of them destroy the feasibility claim, which likely holds in the limited sense that CoTracker can be coaxed into doing stereo matching for textured surfaces like chicken tissue.

This is a paper for people working on endoscopic vision or on novel uses of point-tracking networks. It deserves a serious referee: the idea is novel and the setup is credible, but an editor should ask for the rigid calibration comparison, error bars, and ideally a baseline or two. I'd send it for review, though I wouldn't bet on the exact accuracy numbers surviving contact with a more rigorous evaluation.

Referee Report

4 major / 6 minor

Summary. The paper proposes a markerless 3D tissue tracking method for stereo endoscopic images. It uses two CoTracker3 models: one online model tracks query points over time in the left camera stream, and one offline model performs stereo matching between the left and right frames at each time step. The resulting disparities are converted into 3D positions, with a median over a 3x3 grid of points used as the final estimate. The method is evaluated on a da Vinci laparoscope with a robotic arm moving a 3D-printed phantom and a chicken tissue phantom along randomized trajectories at speeds of 10-80 mm/s. The central quantitative claim is a mean Euclidean error of about 1.1 mm at 10 mm/s on the chicken phantom for full 3D tracking, with real-time performance around 33 FPS.

Significance. The work addresses a relevant problem in surgical navigation: markerless 3D tissue tracking from endoscopic stereo video. The use of an external robot-arm ground truth and a clinical laparoscope is a strength, and the modular combination of temporal and stereo CoTracker models is a reasonable way to capitalize on progress in TAP networks. If the reported millimeter-level accuracy is robust, the method is a promising feasibility demonstration. However, the evaluation currently has several gaps: the coordinate-frame calibration may be absorbing systematic errors if the transformation is fully affine; only three trajectories per speed are used, with means reported without spread; and the point-density hyperparameter appears to be selected on the same data used for the headline results. These issues make the quantitative claim weaker than the abstract suggests.

major comments (4)

[Section 2.2] The manuscript states that 'an affine transformation from camera to robot coordinate system' is fitted using 125 checkerboard positions. If this transformation is a general 3D affine (including scaling and shear), it can compensate for systematic stereo reconstruction errors such as an incorrect baseline or focal length, thereby inflating the apparent tracking accuracy. The reported residual of 0.124 ± 0.062 mm is suspiciously small and the text does not specify whether the residual is a fitting residual or a leave-one-out test error. The authors must state the exact parameterization of the transformation, and re-evaluate the main results using a rigid (rotation+translation) alignment. Without this, the 1.1 mm error reported in Figure 6 cannot be interpreted as the standalone accuracy of the tracking pipeline.
[Section 3, Figures 4-6] The headline errors are mean values over only three randomized trajectories per velocity, with no error bars, per-trajectory values, confidence intervals, or explicit outlier handling. The text mentions that for the 3D-printed phantom 'ignoring these outliers, errors range...', but the outlier definition is never given. With n=3, a single outlier can dominate the mean, and the statement 'there are no outliers' in the XY-plane is not a substitute for a quantitative summary of spread. Please report the full distribution (e.g., median, interquartile range, min/max, or per-run values) and justify the outlier exclusion if the headline 1.1 mm figure is to be credible.
[Section 2.2 and Figure 7] The 3x3 grid (nine query points) is the configuration used in the main evaluation, and Figure 7 shows that this number is near-optimal on the chicken phantom at 50 mm/s. It appears that the point-density experiment and the main accuracy evaluation use the same phantom and similar settings, meaning the hyperparameter is selected on the evaluation data. This is a form of model selection bias. Please clarify which data were used for the density experiments and which for the reported velocities, or better, perform the density experiment on a separate validation set and prespecify the grid size for the main evaluation.
[Section 2.1 and Section 4] The stereo-matching step is a core novel component, but the paper does not provide an isolated evaluation of stereo matching accuracy (e.g., disparity error against a known calibration target) or a comparison to a classical stereo matcher. The Discussion itself concedes that TAP methods like CoTracker are 'not inherently designed for stereo matching, which involves only two images'. Without a component-level analysis or a baseline comparison, it is difficult to attribute the overall tracking error to the temporal tracker versus the stereo matcher, and the claim that TAP-based stereo matching is effective remains under-supported.

minor comments (6)

[Section 3, Figure captions] The figure captions use 'Mean error' but the axis labels only say 'Error'. Please make the unit and the statistic explicit (e.g., 'Mean Euclidean distance error [mm]').
[Section 2.2] The virtual mask used to simulate occluded target boundaries is mentioned but not specified in terms of position, size, opacity, or which trajectories it is applied to. Please provide these details for reproducibility.
[Section 2.2] The three evaluation stages (Z-axis, XY-plane, full 3D) are not fully clear about which pipeline components are active in each stage. For example, in the Z-axis motion stage, is the temporal CoTracker used at all, or is the target position taken directly from the robot? Please clarify.
[Section 3] The term 'outliers' is used without a formal definition. Please define a quantitative criterion (e.g., errors above a threshold or outside a percentile range) and report how many runs were affected.
[Section 4] The claim that the method is 'theoretically capable of handling arbitrarily shaped objects' is not demonstrated by any experiment. Please either provide supporting evidence or tone down the claim.
[General] The paper does not mention code or data availability. For a methods paper in a field that increasingly values reproducibility, a statement on availability (even if 'not available' is stated explicitly) would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline is an empirical adaptation of a pre-trained TAP model, validated against external robot ground truth; no derivation step reduces to its inputs.

full rationale

The paper makes no formal derivation claims. It adapts CoTracker3 (a pre-trained, externally published model) to stereo matching by feeding a left-right pair into a second CoTracker instance, then computes depth from disparity and median-fuses a point ensemble. The only fitted quantities are (a) the affine transformation aligning camera and robot coordinate systems, fitted on 125 checkerboard positions, and (b) the query-point grid size, explored as an ablation. Neither is a 'prediction' in the sense of a derived result: the affine transform is an independent calibration step, and the point-density sweep is a hyperparameter search reported as a curve, not a claim that the optimum was predicted. The headline errors are compared against robot-arm ground truth over trajectories not used to fit any model parameter, so the evaluation is external. The skeptical concern about the affine transform absorbing stereo calibration errors (scaling/shear) is a validity threat to the absolute error numbers, not a circularity: the transform is fit to checkerboard data, not to the tissue trajectories, and the paper does not claim the 0.124 mm residual as a tracking result. The Discussion explicitly concedes that TAP methods are not designed for two-frame stereo matching, which is a limitation statement, not a circular step. There are no self-citations used as load-bearing evidence and no uniqueness theorem or ansatz smuggled via citation. Accordingly, no circular step can be exhibited with the required specificity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical entities, forces, particles, or conserved quantities are postulated. The only newly introduced artifacts are the virtual occlusion mask (an evaluation tool, not an entity) and the two-CoTracker pipeline architecture.

free parameters (2)

Query point grid density (default 3x3 = 9 points) = 9 points (3x3)
Hand-chosen default; Figure 7 shows ~9 points optimal on the chicken phantom, and accuracy degrades beyond ~64 points. This is a configuration choice made from the same evaluation data, not a fitted constant.
Template size for target selection = 100 x 100 pixels
Chosen by the authors from the first frame center; affects which points are tracked but is not optimized or justified beyond being a fixed region of interest.

assumptions (3)

domain assumption The stereo laparoscope produces rectified image pairs with known intrinsic geometry.
Section 2.1 states 'Given a stream of rectified stereo image pairs' and Section 2.2 describes binocular stereo at ~100 mm working distance; disparity to depth conversion assumes rectification and calibration.
domain assumption CoTracker3 correspondence quality transfers from temporal tracking to two-frame stereo matching.
Section 2.1 substitutes a stereo pair for the temporal video input; the discussion concedes TAP models are not inherently designed for stereo matching, which involves only two images. This is the load-bearing transfer assumption.
domain assumption Robot arm end-effector positions logged during motion are an accurate ground truth.
Section 2.2 uses the ABB IRB-120 log as ground-truth while the checkerboard affine calibration carries residual error 0.124 +/- 0.062 mm; no independent verification of robot pose accuracy is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo Images." pith.science (2026). https://pith.science/paper/4P6JUTNW

@misc{pith2026250807851,
  author       = {Pith},
  title        = {Pith review of: Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4P6JUTNW}},
  note         = {Machine review of arXiv:2508.07851}
}

read the original abstract

Minimally invasive surgery presents challenges such as dynamic tissue motion and a limited field of view. Accurate tissue tracking has the potential to support surgical guidance, improve safety by helping avoid damage to sensitive structures, and enable context-aware robotic assistance during complex procedures. In this work, we propose a novel method for markerless 3D tissue tracking by leveraging 2D Tracking Any Point (TAP) networks. Our method combines two CoTracker models, one for temporal tracking and one for stereo matching, to estimate 3D motion from stereo endoscopic images. We evaluate the system using a clinical laparoscopic setup and a robotic arm simulating tissue motion, with experiments conducted on a synthetic 3D-printed phantom and a chicken tissue phantom. Tracking on the chicken tissue phantom yielded more reliable results, with Euclidean distance errors as low as 1.1 mm at a velocity of 10 mm/s. These findings highlight the potential of TAP-based models for accurate, markerless 3D tracking in challenging surgical scenarios.

Figures

Figures reproduced from arXiv: 2508.07851 by the authors.

**Figure 1.** Overview of the proposed method. Eight frames from the left and right camera are received as the input. The tracking points are initialized in the first frame of the left camera. A first CoTracker model is used to track the points along the temporal dimension. Afterwards the frames are stacked as stereo pairs. A second CoTracker model is used to perform stereo matching and find the tracking points in the frames of … view at source ↗

**Figure 3.** Tracking Targets. Left: 3D print with randomly structured surface. Right: Chicken breast. of the checkerboard in each image to calculate an affine transformation from camera to robot coordinate system. We evaluate the system’s performance using a 3D-printed phantom and a chicken tissue phantom. The 3D printed phantom has a structured surface but no distinct features, simulating a specifically challenging tracking … view at source ↗

**Figure 5.** Mean euclidean distance error in mm for movements along the X and Y axis (lateral dimensions) across different velocities [PITH_FULL_IMAGE:figures/full_fig_p003_5.png] view at source ↗

Figures from the paper (2 more)

**Figure 6.** Figure 6: Mean euclidean distance error in mm for 3D movements across different velocities. these outliers, errors range from about 1.1mm for 10 mm/s to around 3.5mm at 80 mm/s [PITH_FULL_IMAGE:figures/full_fig_p003_6.png]

**Figure 7.** Figure 7: analyzes the effect of query point density on accuracy. Initially, increasing the number of points reduces the tracking error, with optimal results achieved around 9 query points. Beyond this point, performance degrades. In runs 2 and 3, configurations with more than …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 5 canonical work pages

[1]

Ricciardi R, et al. The COMPARE study: comparing perioper- ative outcomes of oncologic minimally invasive laparoscopic, Da Vinci robotic, and open procedures: a systematic review and meta-analysis of the evidence. Annals of Surgery. 2025 May 1;281(5):748-63

work page 2025
[2]

Bogani G, et al. Minimally invasive surgery improves short- term outcomes of nerve-sparing radical hysterectomy in pa- tients with cervical cancer: a propensity-matched analysis with open abdominal surgery. Journal of gynecologic oncol- ogy. 2019;30(2)

work page 2019
[3]

Intuitive Annual Report 2024

Guthart G, et al. Intuitive Annual Report 2024. Intuitive Surgi- cal, Inc.; 2025. Available from: https://isrg.intuitive.com/static- files/500ff989-ad91-4b32-a59e-f94a34d75997

work page 2024
[4]

CoTracker3: Simpler and better point tracking by pseudo-labelling real videos

Karaev N, et al. CoTracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.11831. 2024 Oct 15

arXiv 2024
[5]

Deep learning-based stereopsis and monocu- lar depth estimation techniques: a review

Lahiri S, et al. Deep learning-based stereopsis and monocu- lar depth estimation techniques: a review. Vehicles. 2024 Jan 31;6(1):305-51

work page 2024
[6]

A monocular variable magnifications 3D laparo- scope system using double liquid lenses

Mao F , et al. A monocular variable magnifications 3D laparo- scope system using double liquid lenses. IEEE Journal of Translational Engineering in Health and Medicine. 2023 Sep 1;12:32-42

work page 2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Ricciardi R, et al. The COMPARE study: comparing perioper- ative outcomes of oncologic minimally invasive laparoscopic, Da Vinci robotic, and open procedures: a systematic review and meta-analysis of the evidence. Annals of Surgery. 2025 May 1;281(5):748-63

work page 2025

[2] [2]

Bogani G, et al. Minimally invasive surgery improves short- term outcomes of nerve-sparing radical hysterectomy in pa- tients with cervical cancer: a propensity-matched analysis with open abdominal surgery. Journal of gynecologic oncol- ogy. 2019;30(2)

work page 2019

[3] [3]

Intuitive Annual Report 2024

Guthart G, et al. Intuitive Annual Report 2024. Intuitive Surgi- cal, Inc.; 2025. Available from: https://isrg.intuitive.com/static- files/500ff989-ad91-4b32-a59e-f94a34d75997

work page 2024

[4] [4]

CoTracker3: Simpler and better point tracking by pseudo-labelling real videos

Karaev N, et al. CoTracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.11831. 2024 Oct 15

arXiv 2024

[5] [5]

Deep learning-based stereopsis and monocu- lar depth estimation techniques: a review

Lahiri S, et al. Deep learning-based stereopsis and monocu- lar depth estimation techniques: a review. Vehicles. 2024 Jan 31;6(1):305-51

work page 2024

[6] [6]

A monocular variable magnifications 3D laparo- scope system using double liquid lenses

Mao F , et al. A monocular variable magnifications 3D laparo- scope system using double liquid lenses. IEEE Journal of Translational Engineering in Health and Medicine. 2023 Sep 1;12:32-42

work page 2023