REVIEW 4 major objections 5 minor 31 references
First-frame Supervised Video Polyp Segmentation via Propagative and Semantic Dual-teacher Network
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read PSDNet demonstrates that a single annotated frame per clip can train a video polyp segmenter to near-fully-supervised accuracy, by merging a tracker's forward-propagated masks with an EMA teacher's semantic masks using a…
desk verdict A genuinely new task and a clever pseudo-label scoring trick, but the abstract overstates the margin and the scoring's self-reference with XMem is not validated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the back-propagation scoring method. For an unlabeled frame $i$, the propagative teacher produces a mask $p_i$ and the semantic teacher produces $m_i$; each is used to initialize the tracker at frame $i$ and track backward to frame 1, yielding round-tripped masks $e^{1,p_i}$ and $e^{1,m_i}$. Their IoU with the first-frame annotation $y_1$ defines quality scores $C_{p,i}$ and $C_{m,i}$, and the higher-scoring mask becomes the pseudo-label used in the student's cross-entropy loss. This turns the single known label into a temporal consistency check: a good pseudo-label should survive a round trip through the tracker and still align with the known mask.
What would settle it
Compute, on SUN-SEG frames with held-out ground truth, the rank correlation between the backward-propagation IoU score and the true frame-level IoU of each candidate pseudo-label; a non-positive correlation would falsify the claim that the score selects the better teacher output.
Extended reading notes
Core claim
PSDNet's central claim is that one annotated frame per video is enough to train a video polyp segmentation model whose accuracy approaches the fully supervised setting. The propagative teacher supplies temporally coherent masks but drifts over long videos; the semantic teacher is an exponential moving average of the student and supplies time-invariant but less temporally anchored masks. The paper's key move is to use the tracker itself as a judge: for each unlabeled frame, each teacher's candidate mask is propagated backward to the first frame, and the IoU of the round-tripped mask with the first-frame ground truth becomes the quality score. The candidate with the higher score is selected as the pseudo-label for training the student. On SUN-SEG this procedure outperforms all compared sparse-frame supervised methods, with an average 4.5-point Dice gain and at least 3.4 points on every sub-test set, and leaves a gap of only 1.6-3.1 Dice relative to the fully supervised baseline.
Load-bearing premise
The load-bearing premise is that a candidate pseudo-label's quality is faithfully measured by its IoU with the first-frame annotation after a backward trip through the same tracker, and that this test does not systematically favor temporally smooth but spatially wrong masks.
Editorial extensions
If this is right
- First-frame-only supervision is a viable regime for video polyp segmentation: annotation cost drops to one mask per clip (about 1/175 of SUN-SEG's frames) without collapsing accuracy.
- Sparse-frame supervised methods, which annotate dozens of frames per video, are outperformed by a method using strictly fewer labels, by an average 4.5 Dice on SUN-SEG.
- Pseudo-label quality in video self-training can be measured by round-trip consistency with a known anchor, not just by model confidence.
- The same dual-teacher plus backward-scoring setup is expected by the authors to extend to other medical video segmentation tasks with scarce annotations.
Reading between the lines
- The backward-propagation score is a general pseudo-label filter: any video object segmentation method with one reliable reference mask could apply round-trip consistency to reject drifting labels, independent of the polyp domain.
- Since the propagative teacher doubles as the judge, the ceiling of the method is tied to tracker generalization; a tracker fine-tuned on colonoscopy motion and appearance could plausibly close the remaining 1.6-3.1 Dice gap.
- With two or more annotated anchors per video, the same scoring idea could be extended to multi-anchor round trips, which might correct late-frame drift where both teachers degrade.
- A testable scaling prediction follows: if annotation effort is the bottleneck, Dice should improve only slowly as the number of annotated frames grows from one to a few, until tracker or model capacity becomes limiting.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces the First-Frame Supervised Video Polyp Segmentation (FSVPS) task, in which only the first-frame mask of each training video is annotated, and proposes PSDNet, a dual-teacher semi-supervised framework. A propagative teacher (XMem) produces pseudo-labels by forward propagation from the first frame, while a semantic teacher (an EMA copy of the student) produces per-frame segmentations. The two candidate masks are merged by a back-propagation scoring scheme that propagates each candidate backward to the first frame and measures its IoU with the first-frame annotation. The student is trained with a cross-entropy loss on the first-frame annotation and on the merged pseudo-labels. Experiments on SUN-SEG report consistent improvements over four sparse-frame video and four image-based semi-supervised baselines, and Dice within 1.6-3.1 points of the fully supervised upper bound. The paper also compares with zero-shot UVOS models. The central claim is that first-frame-only supervision can approach fully-supervised performance on the largest video polyp segmentation dataset.
Significance. If the results are reproducible, the paper is practically significant: it reduces annotation to one frame per video and reports large gains over existing semi-supervised baselines on SUN-SEG. The FSVPS task formulation is novel and likely to attract follow-up work. The manuscript also releases code, which supports reproducibility. However, the current evidence does not yet establish the mechanism underlying the gains: the back-propagation scorer is not validated as a faithful quality measure, and the empirical claim lacks statistical confidence intervals and a fully specified comparison protocol.
major comments (4)
- [Abstract; Table I] The abstract states that PSDNet improves over sparse-frame supervised state-of-the-arts with a 'minimum improvement of 4.5% in Dice', but Table I shows the smallest per-dataset improvement over the second-best method (TCCNet) is 3.4% on Seen-Easy (0.900 vs. 0.866). The value 4.5% is the average of the four per-dataset improvements. Please correct the abstract to say 'average improvement' or clearly report the aggregation being used, since the current headline claim is factually contradicted by the paper's own table.
- [Section III-C, Table I] All quantitative results in Table I are point estimates from a single run, with no standard deviations, error bars, or significance tests. The reported improvements over the second-best method range from 3.4 to 6.2 Dice points, which could be within run-to-run variance for deep segmentation models. Given the central claim of significant superiority, please report mean +/- std over at least three random seeds and, if feasible, paired significance tests (e.g., per-video Wilcoxon signed-rank or paired t-test) against the best competitor.
- [Section II-C, Eqs. (2)-(3); Table II] The back-propagation quality score is computed by back-propagating the candidate mask with XMem, the same model that generates the propagative teacher's forward pseudo-labels. This creates a systematic bias risk: XMem's backward output is likely to be more consistent with its own forward memory dynamics than with the semantic teacher's segmentation output, so the score may favor the propagative teacher regardless of true segmentation quality. The ablation in Table II demonstrates an overall benefit of adding the back-propagation scorer but does not show that the scorer selects the better teacher per frame. Please validate the scorer by (a) measuring the correlation between the quality score and the IoU of each candidate mask against ground truth on a held-out annotated subset, or (b) comparing PSDNet's automatic selection against an oracle that always selects the candidate with higher ground-truth IoU.
- [Section III-C] The manuscript does not state how many labeled frames are used by each of the eight compared semi-supervised methods. Several of these methods, such as TCCNet and SSTAN, were designed for sparse-frame supervision rather than first-frame-only supervision; if they are given more than one labeled frame per video, the comparison is not apples-to-apples. Please specify the exact annotation budget for each competitor and, if necessary, re-implement them under the first-frame-only setting. Without this information, the headline claim of superiority over sparse-frame supervised state-of-the-arts is not fully established.
minor comments (5)
- [Section I; Section III-A; Section III-B; Section III-E; Conclusion] There are several typos: 'foucs' should be 'focus' in Section I, 'absolution' should be 'absolute' in Section III-A, 'reisze' should be 'resize' in Section III-B, 'Tabel III' should be 'Table III' in Section III-E, and 'FSVOS' in the Conclusion should be 'FSVPS'.
- [Table II] The table uses the symbols '!' and '%' without defining them, and the first row 'None - 0.660' appears incomplete. Please use standard check marks and report all row and column values explicitly.
- [Eq. (2)] The notation 'Propi->1' is confusing because the subscript and arrow suggest a direction opposite to the backward mechanism described in the text. Please clarify the direction of propagation, for example by defining the backward pass explicitly.
- [Table III] The PSDNet column reports a single scalar '0.841 (Test set)' whereas the UVOS baselines appear to be summarized differently. Please report per sub-test set Dice for all methods, and clarify that PSDNet is trained on SUN-SEG while the UVOS baselines are zero-shot prompt-based methods.
- [Section III-B] The EMA weight is set to 0.999, but no sensitivity analysis is provided for this hyperparameter. A short ablation over a few values would strengthen the robustness of the approach.
Circularity Check
No load-bearing circularity: the headline benchmark is externally evaluated, and the only self-referential element is a heuristic scoring bias, not a definitional reduction.
full rationale
The paper's central claim—PSDNet reaches near-fully-supervised performance on SUN-SEG from first-frame-only supervision—is supported by held-out test-set Dice/IoU/MAE numbers computed against ground-truth masks that never enter the pseudo-label merge. The back-propagation scoring (Eqs. 2–3) uses XMem both to generate the propagative teacher's labels and to grade candidates by round-tripping them to the first frame, so the score can be biased toward XMem-consistent masks; this is a real validity risk for the proposed quality measure, but it is not circular because C_{p,i} is not the quantity being predicted and the train-set pseudo-label Dice in Table II is measured against ground truth independently from the scoring rule. Self-citations to the authors' SALI, IBoxCLA, and MonoBox papers are used as backbone or comparison implementations, not as an authority that forces the method; no uniqueness theorem or unverified prior result is imported to forbid alternatives. The FSVPS task name overlaps with semi-supervised video object segmentation, but the contribution is a training framework with an external benchmark, so the naming is not used to derive the reported results. Overall, the derivation chain is externally testable and does not reduce to its inputs by construction.
Assumptions & free parameters
free parameters (1)
- EMA weight =
0.999
assumptions (4)
- domain assumption XMem pretrained on natural videos generalizes to colonoscopy polyp propagation after fine-tuning on synthetic affine clips.
- domain assumption EMA of the student produces stable semantic pseudo labels.
- ad hoc to paper Backward propagation IoU with the first-frame mask measures pseudo-label quality.
- domain assumption SUN-SEG is a representative benchmark for video polyp segmentation.
Cite this review
Pith. "Pith review of First-frame Supervised Video Polyp Segmentation via Propagative and Semantic Dual-teacher Network." pith.science (2026). https://pith.science/paper/YBRISRI5
@misc{pith2026241216503,
author = {Pith},
title = {Pith review of: First-frame Supervised Video Polyp Segmentation via Propagative and Semantic Dual-teacher Network},
year = {2026},
howpublished = {\url{https://pith.science/paper/YBRISRI5}},
note = {Machine review of arXiv:2412.16503}
}
read the original abstract
Automatic video polyp segmentation plays a critical role in gastrointestinal cancer screening, but the cost of frameby-frame annotations is prohibitively high. While sparse-frame supervised methods have reduced this burden proportionately, the cost remains overwhelming for long-duration videos and large-scale datasets. In this paper, we, for the first time, reduce the annotation cost to just a single frame per polyp video, regardless of the video's length. To this end, we introduce a new task, First-Frame Supervised Video Polyp Segmentation (FSVPS), and propose a novel Propagative and Semantic Dual-Teacher Network (PSDNet). Specifically, PSDNet adopts a teacher-student framework but employs two distinct types of teachers: the propagative teacher and the semantic teacher. The propagative teacher is a universal object tracker that propagates the first-frame annotation to subsequent frames as pseudo labels. However, tracking errors may accumulate over time, gradually degrading the pseudo labels and misguiding the student model. To address this, we introduce the semantic teacher, an exponential moving average of the student model, which produces more stable and time-invariant pseudo labels. PSDNet merges the pseudo labels from both teachers using a carefully-designed back-propagation strategy. This strategy assesses the quality of the pseudo labels by tracking them backward to the first frame. High-quality pseudo labels are more likely to spatially align with the firstframe annotation after this backward tracking, ensuring more accurate teacher-to-student knowledge transfer and improved segmentation performance. Benchmarking on SUN-SEG, the largest VPS dataset, demonstrates the competitive performance of PSDNet compared to fully-supervised approaches, and its superiority over sparse-frame supervised state-of-the-arts with a minimum improvement of 4.5% in Dice score.
Figures
Reference graph
Works this paper leans on
-
[1]
Frontiers in intelligent colonoscopy,
G.-P. Ji, J. Liu, P. Xu, N. Barnes, F. S. Khan, S. Khan, and D.-P. Fan, “Frontiers in intelligent colonoscopy,” arXiv preprint arXiv:2410.17241, 2024
arXiv 2024
-
[2]
DACAT: Dual-stream Adaptive Clip-aware Time Modeling for Robust Online Surgical Phase Recognition
K. Yang, Q. Li, and Z. Wang, “Dacat: Dual-stream adaptive clip- aware time modeling for robust online surgical phase recognition,” arXiv preprint arXiv:2409.06217, 2024
work page Pith review arXiv 2024
-
[3]
Pranet: Parallel reverse attention network for polyp segmentation,
D.-P. Fan, G.-P. Ji, T. Zhou, G. Chen, H. Fu, J. Shen, and L. Shao, “Pranet: Parallel reverse attention network for polyp segmentation,” in International conference on medical image computing and computer- assisted intervention. Springer, 2020, pp. 263–273
2020
-
[4]
Learn to threshold: Thresholdnet with confidence-guided manifold mixup for polyp segmentation,
X. Guo, C. Yang, Y . Liu, and Y . Yuan, “Learn to threshold: Thresholdnet with confidence-guided manifold mixup for polyp segmentation,” IEEE transactions on medical imaging , vol. 40, no. 4, pp. 1134–1146, 2020
work page 2020
-
[5]
Shallow attention network for polyp segmentation,
J. Wei, Y . Hu, R. Zhang, Z. Li, S. K. Zhou, and S. Cui, “Shallow attention network for polyp segmentation,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Pro- ceedings, Part I 24 . Springer, 2021, pp. 699–708
work page 2021
-
[6]
Cross- level feature aggregation network for polyp segmentation,
T. Zhou, Y . Zhou, K. He, C. Gong, J. Yang, H. Fu, and D. Shen, “Cross- level feature aggregation network for polyp segmentation,” Pattern Recognition, vol. 140, p. 109555, 2023
work page 2023
-
[7]
Z. Wang, Q. Hu, H. Shi, L. He, M. He, W. Dai, Y . Tian, X. Yang, M. Liu, and Q. Li, “Iboxcla: Towards robust box-supervised segmentation of polyp via improved box-dice and contrastive latent-anchors,” arXiv preprint arXiv:2310.07248, 2023
-
[8]
Monobox: Tightness-free box-supervised polyp segmentation using monotonicity constraint,
Q. Hu, Z. Yi, Y . Zhou, T. Li, F. Huang, M. Liu, Q. Li, and Z. Wang, “Monobox: Tightness-free box-supervised polyp segmentation using monotonicity constraint,” arXiv e-prints, pp. arXiv–2404, 2024
work page 2024
Show all 31 references
-
[9]
Video polyp segmentation: A deep learning perspective,
G.-P. Ji, G. Xiao, Y .-C. Chou, D.-P. Fan, K. Zhao, G. Chen, and L. Van Gool, “Video polyp segmentation: A deep learning perspective,” Machine Intelligence Research, vol. 19, no. 6, pp. 531–549, 2022
2022
-
[10]
An embedding-unleashing video polyp segmentation framework via region linking and scale align- ment,
Z. Fang, X. Guo, J. Lin, H. Wu, and J. Qin, “An embedding-unleashing video polyp segmentation framework via region linking and scale align- ment,” in Proceedings of the AAAI conference on artificial intelligence , vol. 38, no. 2, 2024, pp. 1744–1752
2024
-
[11]
Sali: Short- term alignment and long-term interaction network for colonoscopy video polyp segmentation,
Q. Hu, Z. Yi, Y . Zhou, F. Peng, M. Liu, Q. Li, and Z. Wang, “Sali: Short- term alignment and long-term interaction network for colonoscopy video polyp segmentation,” arXiv preprint arXiv:2406.13532 , 2024
2024 arXiv
-
[12]
Perturbed and strict mean teachers for semi-supervised semantic seg- mentation,
Y . Liu, Y . Tian, Y . Chen, F. Liu, V . Belagiannis, and G. Carneiro, “Perturbed and strict mean teachers for semi-supervised semantic seg- mentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 4258–4267
2022
-
[13]
Acl-net: semi-supervised polyp segmentation via affinity contrastive learning,
H. Wu, W. Xie, J. Lin, and X. Guo, “Acl-net: semi-supervised polyp segmentation via affinity contrastive learning,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 37, no. 3, 2023, pp. 2812–2820
2023
-
[14]
Corrmatch: Label propagation via correlation matching for semi-supervised semantic segmentation,
B. Sun, Y . Yang, L. Zhang, M.-M. Cheng, and Q. Hou, “Corrmatch: Label propagation via correlation matching for semi-supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , 2024, pp. 3097–3107
2024
-
[15]
Tccnet: Temporally consistent context-free network for semi- supervised video polyp segmentation
X. Li, J. Xu, Y . Zhang, R. Feng, R.-W. Zhao, T. Zhang, X. Lu, and S. Gao, “Tccnet: Temporally consistent context-free network for semi- supervised video polyp segmentation.” in IJCAI, 2022, pp. 1109–1115
2022
-
[16]
Semi- supervised spatial temporal attention network for video polyp segmen- tation,
X. Zhao, Z. Wu, S. Tan, D.-J. Fan, Z. Li, X. Wan, and G. Li, “Semi- supervised spatial temporal attention network for video polyp segmen- tation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2022, pp. 456–466
2022
-
[17]
Semi-supervised video semantic segmentation with inter-frame feature reconstruction,
J. Zhuang, Z. Wang, and Y . Gao, “Semi-supervised video semantic segmentation with inter-frame feature reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 3263–3271
2022
-
[18]
Infer from what you have seen before: Temporally-dependent classifier for semi-supervised video segmentation,
J. Zhuang, Z. Wang, Y . Zhang, and Z. Fan, “Infer from what you have seen before: Temporally-dependent classifier for semi-supervised video segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 3575–3584
2024
-
[19]
Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model,
H. K. Cheng and A. G. Schwing, “Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model,” in European Conference on Computer Vision . Springer, 2022, pp. 640–658
2022
-
[20]
Segment and track anything,
Y . Cheng, L. Li, Y . Xu, X. Li, Z. Yang, W. Wang, and Y . Yang, “Segment and track anything,” arXiv preprint arXiv:2305.06558 , 2023
2023 arXiv
-
[21]
Track anything: Segment anything meets videos,
J. Yang, M. Gao, Z. Li, S. Gao, F. Wang, and F. Zheng, “Track anything: Segment anything meets videos,” arXiv preprint arXiv:2304.11968 , 2023
2023 arXiv
-
[22]
Sam 2: Segment anything in images and videos,
N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R¨adle, C. Rolland, L. Gustafson et al., “Sam 2: Segment anything in images and videos,” arXiv preprint arXiv:2408.00714 , 2024
2024 arXiv
-
[23]
The 2017 davis challenge on video object segmen- tation,
J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbel ´aez, A. Sorkine-Hornung, and L. Van Gool, “The 2017 davis challenge on video object segmen- tation,” arXiv preprint arXiv:1704.00675 , 2017
2017 arXiv
-
[24]
Youtube-vos: A large-scale video object segmentation benchmark,
N. Xu, L. Yang, Y . Fan, D. Yue, Y . Liang, J. Yang, and T. Huang, “Youtube-vos: A large-scale video object segmentation benchmark,” arXiv preprint arXiv:1809.03327 , 2018
2018 arXiv
-
[25]
Pytorch: An imperative style, high-performance deep learning library,
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems , vol. 32, 2019
2019
-
[26]
Pvt v2: Improved baselines with pyramid vision transformer,
W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pvt v2: Improved baselines with pyramid vision transformer,” Computational Visual Media , vol. 8, no. 3, pp. 415–424, 2022
2022
-
[27]
Imagenet: A large-scale hierarchical image database,
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255
2009
-
[28]
Adam: A method for stochastic optimization,
D. P. Kingma, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[29]
A novel parallel cooperative mean- teacher framework (pcmt) combined with prediction uncertainty guide and class contrastive learning for semi-supervised polyp segmentation,
Y . Xia, H. Yun, P. Liu, and M. Li, “A novel parallel cooperative mean- teacher framework (pcmt) combined with prediction uncertainty guide and class contrastive learning for semi-supervised polyp segmentation,” Expert Systems with Applications , vol. 255, p. 124816, 2024
2024
-
[30]
Segment anything in medical images and videos: Benchmark and deployment,
J. Ma, S. Kim, F. Li, M. Baharoon, R. Asakereh, H. Lyu, and B. Wang, “Segment anything in medical images and videos: Benchmark and deployment,” arXiv preprint arXiv:2408.03322 , 2024
2024 arXiv
-
[31]
Medical sam 2: Segment medical images as video via segment anything model 2,
J. Zhu, Y . Qi, and J. Wu, “Medical sam 2: Segment medical images as video via segment anything model 2,” arXiv preprint arXiv:2408.00874, 2024
2024 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.