Pith. sign in

REVIEW 4 major objections 5 minor 19 references

You Are Your Best Teacher: Semi-Supervised Surgical Point Tracking with Cycle-Consistent Self-Distillation

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A point tracker can adapt to surgical video by learning from its own filtered output.

desk verdict A plausible semi-supervised adaptation study with a genuinely new single-teacher finding, but test-set tuning and missing error bars make the exact gains uncertain. read the letter →

arxiv 2505.05722 v1 pith:7CPGNDC7 submitted 2025-05-09 cs.CV

Valay Bundele , Mehran Hosseinzadeh , Hendrik Lensch This is my paper · ORCID
classification cs.CV
keywords surgicalpointtrackingsemi-supervisedlearningcycleconsistencyself-distillationpseudo-labelfilteringdomainadaptationCholec80STIRbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a point tracker trained on synthetic data can be adapted to surgical video without any manual annotations, using only 80 unlabeled videos and a single frozen copy of the model as its own teacher. The proposed framework, SurgTracker, generates pseudo-trajectories on unlabeled surgical footage and keeps only the ones that survive a forward-then-backward tracking check, then fine-tunes the student on those filtered labels. If the claim holds, it matters because manual point-level annotation in surgery is costly and impractical, and methods that work on natural video—teacher ensembles or heavy augmentation—have not been shown to survive the larger domain shift of surgical scenes. On the STIR benchmark the paper reports that SurgTracker lowers mean endpoint error from 17.01 to 16.27 relative to its teacher and baseline, CoTracker3, and beats all compared trackers.

What carries the argument

The central mechanism is the cycle-consistency filter defined by Eq. (1), paired with a fixed teacher identical to the student. For each query point, the frozen teacher tracks forward to the end of the sequence, then tracks backward from the predicted endpoint to the start; the Euclidean distance between the original query point and the backward-tracked endpoint is the cycle error. Trajectories whose error is below $\alpha = 5$ are kept as pseudo-labels for student fine-tuning. Because the teacher never updates, the supervision signal is stable across training batches, and because teacher and student share architecture and initialization, the student is asked to imitate a representation it already contains. The threshold $\alpha$ is the main dial controlling the trade-off between label quantity and label quality.

What would settle it

On the STIR ground-truth annotations, compute for every teacher pseudo-trajectory both the cycle error of Eq. (1) and the true endpoint error against ground truth, and check whether trajectories with cycle error below $\alpha = 5$ have systematically smaller true error than those above it. If the correlation is weak, or if many low-cycle-error trajectories are far from ground truth, the filtering premise fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that filtered self-distillation from a single architecture-aligned teacher is enough to adapt a synthetic-pretrained point tracker to a high-shift domain. SurgTracker initializes both teacher and student from CoTracker3 (Online), freezes the teacher, and uses it to predict trajectories on 80 unlabeled Cholec80 videos. A trajectory is retained only if its cycle error—the Euclidean distance between the original query point and the point recovered by tracking forward and then backward—is below a threshold $\alpha = 5$. The student is then fine-tuned with a Huber tracking loss that pays more attention to visible points and later refinement iterations. The reported results on STIR are MEE 16.27 versus 17.01, MCD 17.12 versus 17.81, and average accuracy 68.55 versus 68.11 against CoTracker3, with ablations showing that removing the filter hurts and that a single teacher beats multi-teacher ensembles.

Load-bearing premise

The load-bearing assumption is that the frozen teacher's own cycle error—how close a point comes back to where it started after tracking forward and then backward—tells which forward trajectories are trustworthy; if the teacher fails symmetrically or occlusions break the round trip in misleading ways, the filter will keep bad pseudo-labels or throw away good ones.

Editorial extensions

If this is right

  • If the central claim is correct, surgical point tracking can be improved with unlabeled footage alone, removing the annotation bottleneck that currently limits tissue and tool tracking.
  • A single consistent teacher can outperform teacher ensembles under large domain shift, so the extra cost and complexity of multi-teacher pseudo-labeling may not be justified in high-shift domains.
  • Cycle-consistency filtering is doing real work: the ablations show that unfiltered pseudo-labels degrade accuracy on the STIR benchmark.
  • Because the teacher is frozen and only one model is kept in memory, the adaptation pipeline is cheaper than ensemble-based alternatives, which matters for real-time clinical deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the same single-teacher cycle-consistent recipe transfers to other high-shift, label-scarce domains such as endoscopy, ultrasound, or microscopy; that is a natural next test.
  • The reported gains over CoTracker3 are modest (0.74 pixels in mean endpoint error), so the practical value for downstream clinical tasks will depend on whether endpoint errors of this size change decisions; the paper does not test that.
  • Because query points are sampled with SIFT, supervision is concentrated on textured, trackable tissue; low-texture structures such as smooth organs or specular regions may receive few pseudo-labels, and the paper does not measure per-region accuracy.
  • The same cycle-consistency principle could be applied at test time to re-score or repair trajectories, not just to filter training labels; the paper only uses it during training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SurgTracker, a semi-supervised framework that adapts the synthetic-pretrained point tracker CoTracker3 to surgical videos using only unlabeled Cholec80 videos. Pseudo-labels are generated online by a frozen teacher that is architecturally identical and identically initialized to the student, and are filtered by a cycle-consistency check (Eq. 1) that discards trajectories whose forward-backward endpoint error exceeds a threshold α. The student is then fine-tuned with a Huber tracking loss. Experiments on the STIR benchmark report that SurgTracker outperforms CoTracker3 and other baselines on MEE, MCD, and <δx_avg, with MEE improving from 17.01 to 16.27. Ablations show that cycle-consistency filtering helps and that a single teacher outperforms multi-teacher supervision in this domain.

Significance. If the reported results hold, the paper makes a useful empirical contribution: it demonstrates that self-distillation with a single architecture-aligned teacher plus cycle-consistency filtering can adapt a synthetic-pretrained point tracker to a high-shift, low-annotation domain such as surgery. The work is also relevant to the broader semi-supervised point-tracking literature because it challenges the assumption that teacher diversity is necessary for pseudo-label quality, and it provides an ablation (Table 3) showing that a single teacher outperforms multi-teacher configurations on this benchmark. The method is simple and computationally efficient, and the unfiltered self-distillation result (16.69 MEE) still beats the teacher baseline (17.01 MEE), which suggests the core idea is not entirely dependent on the filtering threshold. However, the evaluation protocol currently weakens confidence in the exact reported numbers: the cycle threshold is selected on the STIR test set, approximately 20 test sequences are excluded post hoc, and no error bars or multiple-seed results are provided. The paper does not release code or checkpoints, which limits reproducibility.

major comments (4)
  1. [§4.2, §4.4, Table 2] The cycle-consistency threshold α is selected using the STIR benchmark itself: §4.2 states α=5 'provides the best trade-off,' and Table 2 is an ablation computed on STIR. Because the reported headline gain over CoTracker3 is only 0.74 pixels MEE, tuning α on the test set can materially inflate this gain. The authors should either select α on a held-out validation split of STIR (or on a separate validation set) and report the corresponding test numbers, or clearly state that all Table 1/2 numbers are exploratory and provide a corrected comparison with α fixed before seeing STIR.
  2. [§4.1] The evaluation excludes 'around 20 sequences with excessive label noise' from the STIR benchmark, but the manuscript does not specify the exact number, the criterion used to identify these sequences, or whether the exclusion was decided before or after inspecting model predictions. Post hoc test-set filtering can bias the comparison in favor of the proposed method. The authors should report the precise filtering rule, list or characterize the excluded sequences, and provide results with and without the exclusion to show robustness.
  3. [§4.3, Table 1] No error bars, confidence intervals, or multiple-seed experiments are reported for any of the tables. The reported MEE improvement over CoTracker3 is 0.74 pixels (about 4.3% relative), which may be within run-to-run or sequence-level noise given that the training procedure involves random sequence sampling, random strides, and stochastic optimization. The authors should report standard deviations over at least three seeds or bootstrap confidence intervals over test sequences, and ideally release code or checkpoints so the community can verify the comparison.
  4. [§3.2.2, Eq. (1)] The cycle-consistency filter checks only the Euclidean distance between the original query point and the endpoint of the backward track. This is a weak proxy for pseudo-label quality: a forward trajectory and a backward trajectory can both lock onto the same wrong but temporally coherent point and still pass the endpoint check, and reverse tracking in the presence of occlusion may produce unreliable endpoints. Table 2 shows that even without filtering SurgTracker improves over CoTracker3, so the headline claim does not rest solely on this filter, but the paper's central design rationale does. The authors should analyze filter quality (e.g., precision of kept versus discarded pseudo-labels on a labeled validation set, or a study of failure cases) to substantiate the claim that cycle consistency selects high-quality trajectories.
minor comments (5)
  1. [Title and abstract] There is a spacing artifact in the title ('Y ou Are Y our Best Teacher') and the abstract contains the same issue; this should be corrected.
  2. [§3.2.3, Eq. (2)] The loss formulation uses a Huber threshold of 6 and an occlusion down-weighting factor of 1/5, but the paper does not state whether these match CoTracker3 or are newly introduced; please clarify the provenance of these hyperparameters.
  3. [§4.1] The description of the STIR benchmark says it comprises 'around 425' videos and annotations of 'over 3,000 points,' but the exact split sizes and the number of sequences used after filtering are not given; please provide exact counts.
  4. [Figure 2] The qualitative figure caption mentions red, green, blue, and pink markers, but the text does not explain what the green dots and pink lines represent; please make the caption self-contained.
  5. [§4.4, Table 2] The differences between α=2.5 and α=7.5 are small (MEE 16.76 and 16.43 vs. 16.27 at α=5); the text should discuss whether these differences are meaningful given the absence of uncertainty estimates.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the reported STIR improvement is an external-benchmark empirical result, not a construction-level tautology.

full rationale

The paper's chain is empirical rather than derivational: pseudo-labels are produced by a frozen teacher on unlabeled Cholec80 video, filtered by the cycle-consistency check of Eq. 1, and used to fine-tune a student that is initialized from the same teacher. The claimed improvement is then measured against ground-truth point tracks on the STIR benchmark, which is a different dataset from the unlabeled training corpus. The teacher/student identity makes the supervision self-referential, but it does not make the evaluation a tautology: STIR annotations are external, and the training loss (Eq. 2) is not algebraically identical to the evaluation metrics MEE/MCD/<delta_x. No load-bearing self-citation appears; all cited methods are external prior work. The choice of the cycle threshold alpha=5 from a STIR ablation and the exclusion of about 20 noisy STIR sequences are evaluation-protocol risks (test-set tuning and post hoc filtering) that bear on validity and reproducibility, but they do not exhibit a specific reduction of the claimed result to its inputs by construction. Hence no circular step can be quoted, and the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. Its free parameters and assumptions are standard for semi-supervised domain adaptation, but the test-set tuning of alpha and the unvalidated transfer assumption weaken the contribution.

free parameters (1)
  • cycle consistency threshold alpha = 5
    Selected as the best trade-off in Table 2 on the STIR test set; tuned on the evaluation benchmark, which inflates reported performance.
assumptions (4)
  • domain assumption The frozen teacher's forward/backward cycle error is a valid proxy for pseudo-label quality (Eq. 1, Sec 3.2.2).
    The entire filtering mechanism rests on this assumption; if cycle error does not correlate with true tracking error, the method has no principled way to select good labels.
  • domain assumption SIFT keypoints in the first frame provide trackable points relevant for surgical tracking.
    Used in Sec 3.2.1; if keypoints are not on tissue or are unstable, the pseudo-labels are poor regardless of filtering.
  • domain assumption Fine-tuning on Cholec80 laparoscopic videos transfers to the STIR robotic surgery benchmark.
    Training and evaluation domains differ (laparoscopic vs. da Vinci robotic); the paper does not validate on same-domain surgical data.
  • domain assumption CoTracker3's pretrained weights provide a good starting point for surgical tracking.
    The teacher and student are initialized from CoTracker3; if the source model is too far from the target domain, self-distillation cannot recover.

how reviews work

0 comments
Cite this review

Pith. "Pith review of You Are Your Best Teacher: Semi-Supervised Surgical Point Tracking with Cycle-Consistent Self-Distillation." pith.science (2026). https://pith.science/paper/7CPGNDC7

@misc{pith2026250505722,
  author       = {Pith},
  title        = {Pith review of: You Are Your Best Teacher: Semi-Supervised Surgical Point Tracking with Cycle-Consistent Self-Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7CPGNDC7}},
  note         = {Machine review of arXiv:2505.05722}
}
read the original abstract

Synthetic datasets have enabled significant progress in point tracking by providing large-scale, densely annotated supervision. However, deploying these models in real-world domains remains challenging due to domain shift and lack of labeled data-issues that are especially severe in surgical videos, where scenes exhibit complex tissue deformation, occlusion, and lighting variation. While recent approaches adapt synthetic-trained trackers to natural videos using teacher ensembles or augmentation-heavy pseudo-labeling pipelines, their effectiveness in high-shift domains like surgery remains unexplored. This work presents SurgTracker, a semi-supervised framework for adapting synthetic-trained point trackers to surgical video using filtered self-distillation. Pseudo-labels are generated online by a fixed teacher-identical in architecture and initialization to the student-and are filtered using a cycle consistency constraint to discard temporally inconsistent trajectories. This simple yet effective design enforces geometric consistency and provides stable supervision throughout training, without the computational overhead of maintaining multiple teachers. Experiments on the STIR benchmark show that SurgTracker improves tracking performance using only 80 unlabeled videos, demonstrating its potential for robust adaptation in high-shift, data-scarce domains.

Figures

Figures reproduced from arXiv: 2505.05722 by the authors.

Figure 1
Figure 1. Overview of the SurgTracker framework. Given an unlabeled surgical video, pseudo-labels are generated by a frozen teacher [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison of CoTracker3 and our method on a chal [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 14 canonical work pages

  1. [1]

    Track-on: Transformer-based online point tracking with memory

    G ¨orkay Aydemir, Xiongyi Cai, Weidi Xie, and Fatma G¨uney. Track-on: Transformer-based online point tracking with memory. arXiv preprint arXiv:2501.18487, 2025. 2

    arXiv 2025

  2. [2]

    Local all-pair correspon- dence for point tracking

    Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seun- gryong Kim, and Joon-Young Lee. Local all-pair correspon- dence for point tracking. In European Conference on Com- puter Vision, pages 306–325. Springer, 2024. 1, 2

    work page 2024

  3. [3]

    Tap-vid: A benchmark for track- ing any point in a video

    Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Re- casens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for track- ing any point in a video. Advances in Neural Information Processing Systems, 35:13610–13626, 2022. 3

    2022

  4. [4]

    Tapir: Tracking any point with per-frame initialization and temporal refinement

    Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 10061– 10072, 2023. 1, 2, 4

    work page 2023

  5. [5]

    Bootstap: Bootstrapped training for tracking-any-point

    Carl Doersch, Pauline Luc, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Rocco, Ross Goroshin, Jo˜ao Carreira, et al. Bootstap: Bootstrapped training for tracking-any-point. In Proceedings of the Asian Conference on Computer Vision, pages 3257–3274, 2024. 1, 2, 4

    work page 2024

  6. [6]

    Particle video revisited: Tracking through occlusions using point trajectories

    Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vi- sion, pages 59–75. Springer, 2022. 2

    2022

  7. [7]

    Patient-Specific Domain Adaptation for Fast Optical Flow Based on Teacher-Student Knowledge Transfer

    Sontje Ihler, Max-Heinrich Laves, and Tobias Ortmaier. Patient-specific domain adaptation for fast optical flow based on teacher-student knowledge transfer. arXiv preprint arXiv:2007.04928, 2020. 1, 2

    work page Pith review arXiv 2007

  8. [8]

    Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos

    Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos. arXiv preprint arXiv:2410.11831 ,

    arXiv

Show all 19 references
  1. [9]

    Co- tracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker: It is better to track together. InEuropean Conference on Computer Vision, pages 18–35. Springer, 2024. 1, 2

    2024

  2. [10]

    Taptr: Tracking any point with transformers as detection

    Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, and Lei Zhang. Taptr: Tracking any point with transformers as detection. In European Confer- ence on Computer Vision, pages 57–75. Springer, 2024. 1

    2024

  3. [11]

    Object recognition from local scale-invariant features

    David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE interna- tional conference on computer vision , pages 1150–1157. Ieee, 1999. 3

    1999

  4. [12]

    Sendd: Sparse efficient neural depth and deformation for tissue tracking

    Adam Schmidt, Omid Mohareri, Simon DiMaio, and Septi- miu E Salcudean. Sendd: Sparse efficient neural depth and deformation for tissue tracking. In International Conference on Medical Image Computing and Computer-Assisted Inter- vention, pages 238–248. Springer, 2023. 2, 4

    2023

  5. [13]

    Tracking and mapping in medical computer vision: A review.Medical Image Analysis, page 103131, 2024

    Adam Schmidt, Omid Mohareri, Simon DiMaio, Michael C Yip, and Septimiu E Salcudean. Tracking and mapping in medical computer vision: A review.Medical Image Analysis, page 103131, 2024. 1

    2024

  6. [14]

    Refining pre-trained motion models

    Xinglong Sun, Adam W Harley, and Leonidas J Guibas. Refining pre-trained motion models. In 2024 IEEE Inter- national Conference on Robotics and Automation (ICRA) , pages 4932–4938. IEEE, 2024. 2

    2024

  7. [15]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  8. [16]

    En- donet: a deep architecture for recognition tasks on laparo- scopic videos

    Andru P Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. En- donet: a deep architecture for recognition tasks on laparo- scopic videos. IEEE transactions on medical imaging , 36 (1):86–97, 2016. 3

    2016

  9. [17]

    Tracking everything everywhere all at once

    Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 19795–19806, 2023. 1, 2

    2023

  10. [18]

    Tracking everything in robotic-assisted surgery

    Bohan Zhan, Wang Zhao, Yi Fang, Bo Du, Francisco Vas- concelos, Danail Stoyanov, Daniel S Elson, and Baoru Huang. Tracking everything in robotic-assisted surgery. arXiv preprint arXiv:2409.19821, 2024. 1, 2, 3

    2024 arXiv

  11. [19]

    Pointodyssey: A large-scale synthetic dataset for long-term point tracking

    Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wet- zstein, and Leonidas J Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 19855–19865, 2023. 2

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.