REVIEW 4 major objections 6 minor 35 references
Part Segmentation for Highly Accurate Deformable Tracking in Occlusions via Fully Convolutional Neural Networks
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Restricting point-cloud associations to pixels labelled by a fast fully convolutional network keeps a geometric 3D human tracker accurate when objects occlude the body.
desk verdict A practical hybrid RGB-D tracker with a real label-constrained filtering idea, but the accuracy claim is weakened by a self-referential evaluation that uses the baseline tracker as ground truth. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Fast-FCN, a fully convolutional network with nine convolution layers, three max-pooling layers, and three deconvolutional layers, operating on 128x106 four-channel RGB-D input and outputting seven labels: background, head, torso, left arm, right arm, left leg, and right leg. During tracking it supplies a semantic label for every pixel, and the tracker, based on the window-based data association of [3], only associates model vertices with observed points carrying the same label. The same map also yields approximate centroids per body part that inject a low-weight loss term during initialization and tracking recovery, driving the model within about 20 centimeters so vertex-based optimization can take over.
What would settle it
Take unoccluded RGB-D sequences with independently measured ground-truth pose, train Fast-FCN on that ground truth instead of on the tracker's output, and compare joint error against the paper's pipeline; if the improvement over the geometric tracker disappears when labels no longer come from the tracker, the reported gain is partly an artifact of the tracker serving as its own ground truth.
Extended reading notes
Core claim
The central discovery is that semantic part segmentation becomes a gate for data association: restricting model-vertex-to-point-cloud correspondences to pixels with matching body-part labels prevents occluders and crossed body parts from pulling the optimization away from the true pose. The paper further shows that this filtering, combined with a segmentation-based initialization loss, keeps 3D joint accuracy high when objects are inserted 30, 45, or 60 cm from the person, where the geometric tracker alone loses track. The training procedure is also part of the discovery: an existing geometric tracker labels unoccluded frames, artificial objects are overlaid to synthesize occlusion, and the resulting maps train Fast-FCN without manual annotation. The paper also reports that a model trained without occlusion augmentation produces segmentation masks that hurt tracking, so the augmented data generation is a necessary part of the pipeline.
Load-bearing premise
The geometric tracker [3] is accurate enough on unoccluded frames to serve as both the training-label source and the evaluation ground truth; if that tracker carries systematic bias, the bias enters both the FCN's labels and the measured accuracy, so the reported improvement would not prove true 3D accuracy.
Editorial extensions
If this is right
- Filtering point-cloud associations by semantic label should make model-based RGB-D trackers robust to nearby occluding objects, because occluder points are labelled background and never enter the association.
- Because the FCN labels are generated by an existing tracker plus synthetic occlusion, the pipeline can be deployed in new settings with almost no manual annotation cost.
- The Fast-FCN runs at 7.86 milliseconds per frame, so semantic filtering can run alongside geometric optimization while keeping interactive frame rates.
- Training the segmenter without occlusion data makes tracking worse than the geometric tracker alone, so the augmented-data generation is a required element of the method.
- Raw segmentation metrics such as per-part intersection over union are not strongly indicative of final 3D tracking accuracy, so hybrid systems should be evaluated on joint-position error.
Reading between the lines
- The same label-gated association could be extended beyond body parts to object labels, letting a robot track a person who is holding or partially hidden by a known object.
- The synthetic-occlusion training strategy suggests a self-improvement loop: whenever the tracker succeeds on a hard frame, that frame could be fed back as training data, progressively expanding the set of occluders the system handles without manual labels.
- The paper tests single-object occlusion covering about one-third of the body; heavier occlusion or multiple simultaneous occluders would likely reveal the failure point of the label-gated association.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a hybrid RGB-D human pose tracking system that couples a lightweight fully convolutional network (Fast-FCN) with an optimization-based geometric tracker. The FCN performs pixel-wise semantic segmentation of body parts, and the tracker restricts point-cloud data association to pixels whose semantic label matches the model vertex, thereby filtering out clutter and occluding objects. Additionally, the segmentation is used to generate a centroid-based initialization/recovery loss. Training data are generated automatically by running the geometric tracker on unoccluded videos and inserting artificial occluding objects, avoiding manual annotation. The authors evaluate the system on a small holdout set of augmented sequences and report tracking accuracy versus distance thresholds, comparing against the baseline geometric tracker and against U-Net and VGG-FCN for segmentation quality. The main claimed contribution is that semantic filtering improves 3D tracking accuracy under occlusion while maintaining interactive frame rates.
Significance. If the reported results were externally validated, the paper would make a useful practical contribution: it demonstrates a real-time hybrid approach in which a fast FCN segmenter improves the robustness of a model-based RGB-D tracker under occlusion, and it shows that training labels can be produced nearly automatically via self-supervision from the tracker itself plus data augmentation. The architecture comparison (Table I) is a strength: the proposed Fast-FCN is substantially faster than U-Net and VGG-FCN while achieving comparable segmentation IoU. The automatic labeling pipeline and the explicit ablation of augmentation (Section V-B) are also valuable. However, the significance is currently limited by an evaluation that is thin and partly circular, as detailed below. The central idea is plausible, but the evidence as presented does not yet establish the claimed high 3D accuracy.
major comments (4)
- [Section IV-A and V-A] The ground truth for both the FCN training labels and the evaluation of tracking accuracy is generated by the same geometric tracker [3] that is used as the baseline comparison. Section IV-A states that tracker [3] was applied to unoccluded sequences, with manual discarding of visibly failed frames, to produce ground-truth labels. If [3] has systematic bias on those sequences, the bias enters both the training labels and the evaluation reference, so the improvement reported in Figures 4 and 6 may reflect better agreement with [3]'s internal body-model assumptions rather than true 3D accuracy. No independent motion-capture, manual-annotation, or synthetic-known-pose ground truth is provided on the holdout sequences, so the absolute accuracy claim is not externally validated.
- [Section V-A and Figure 4] The quantitative tracking evaluation is based on only four holdout sequences and reports no error bars, per-sequence breakdown, or statistical tests. The abstract and Section V-A claim "significant improvement," but without any measure of variance or a significance test, this claim is not supported. Please report per-sequence accuracy curves, error bars or confidence intervals, and, if appropriate, a paired statistical test across the four sequences.
- [Section V and Figure 6] The comparison with the state-of-the-art regression method [33] is only qualitative: Figure 6 shows example frames, but no quantitative tracking accuracy numbers are provided for [33] on the same object-inserted and human-object interaction sequences. Since the central claim is about 3D tracking accuracy, the reader cannot assess whether the proposed system actually outperforms [33]. Add quantitative results for [33] on the same evaluation protocol.
- [Section III-C and V-B] The method introduces several free parameters — the class-imbalance weight lambda, the centroid loss weight, and the 20 cm initialization distance threshold — but the paper reports no sensitivity analysis. Section V-B demonstrates that a poorly trained segmenter can degrade tracking relative to the geometric tracker alone, which shows that segmentation quality is consequential for the final accuracy. Please report how these parameters were selected and provide some sensitivity results to show that the reported improvement is not restricted to a fragile parameter setting.
minor comments (6)
- [Throughout] There are several typos: "descriminative" in the Introduction, "perfomance" in Section II, "segementation" in Section V, and "Regognition" in reference [33].
- [Section IV-B and V-A] Section IV-B says inserted objects occlude "around one-third to a half" of the person, while Section V-A says the test inserted an object occluding "around one third" of the body; these descriptions should be reconciled.
- [Section IV and V] The relationship between the 32 unoccluded sequences, the 7 human-object interaction sequences, the 28 training sequences, and the 4 holdout sequences is unclear; specify exactly which sequences and which augmentation objects are used for training, validation, and testing.
- [Section III-A] Training details for the Fast-FCN — number of epochs, learning rate, batch size, and optimizer hyperparameters — are not reported, which limits reproducibility.
- [Figures 4, 5, and 7] The y-axis label "Accuracy (%)" is ambiguous; consider "proportion of joints within threshold" and clarify the meaning of the multiple curves (e.g., 30 cm, 45 cm, 60 cm distance thresholds) in the x-axis label.
- [Tables II and III] The segmentation metrics are reported without stating the number of test frames used for evaluation; including this information would help interpret the numbers.
Circularity Check
The central accuracy comparison is scored against ground truth produced by the same geometric tracker that serves as the baseline and as the FCN's label source, so the reported gain may measure self-consistency rather than independent 3D accuracy.
-
fitted input called prediction
[Section IV-A (Data Generation) through Section V-A (Tracking accuracy), Figure 4.]
"We applied the tracking system in [3] to these videos and used the output to generate ground truth for each video frame. ... After running each sequence, we manually discarded any frames where the tracker failed, which was the only manual process in this procedure. ... We first compare our method with the tracking system proposed in [3]."
The tracker [3] simultaneously provides the training labels for Fast-FCN, is the baseline algorithm in the comparison, and generates the ground-truth joint positions against which both methods are scored. Consequently, the accuracy curves in Figure 4 measure how close each tracker is to [3]'s own unoccluded pose estimates, not to an independent 3D reference. Any systematic bias in [3] enters both the supervision signal and the evaluation truth, so the reported improvement over [3] can be inflated by self-consistency; the FCN is rewarded for reproducing [3]'s body-model assumptions rather than for verified true 3D accuracy.
full rationale
The only load-bearing circularity is in the evaluation of the paper's main claim: the ground-truth joint positions are generated by the same tracker [3] that is the baseline and the source of FCN training labels. This makes the reported 'significant improvement' a self-referential comparison: it shows that the FCN-filtered tracker stays closer to [3]'s unoccluded output under occlusion than [3] itself does, but it does not establish absolute 3D accuracy against an external reference. This is not a full derivation collapse, because the test-time occlusion is new and the FCN's semantic filtering could still improve tracking independently; however, the absence of independent ground truth on the key test set means the central quantitative claim is partially forced by the shared label source. The architecture comparisons in Section V-C (Table II, Table III, Figure 7) and the data-augmentation comparison in Section V-B are self-contained and not circular, since they compare segmentation models on segmentation metrics and a fixed training regime. We therefore assign a score of 6, reflecting partial circularity in the main accuracy evaluation rather than a completely self-referential derivation.
Assumptions & free parameters
free parameters (3)
- lambda (class imbalance weight)
- centroid loss weight
- initialization distance threshold =
20 cm
assumptions (4)
- domain assumption Tracker [3] produces accurate poses and part labels on unoccluded sequences.
- domain assumption Overlaying object RGB-D images onto frames is a valid simulation of real-world occlusion.
- domain assumption Pixel-wise semantic labels from the FCN correspond to the model's 3D part vertices.
- standard math Standard deep learning operations (convolution, softmax, cross-entropy) behave as expected.
Cite this review
Pith. "Pith review of Part Segmentation for Highly Accurate Deformable Tracking in Occlusions via Fully Convolutional Neural Networks." pith.science (2026). https://pith.science/paper/7O6B5PQM
@misc{pith2026190801504,
author = {Pith},
title = {Pith review of: Part Segmentation for Highly Accurate Deformable Tracking in Occlusions via Fully Convolutional Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/7O6B5PQM}},
note = {Machine review of arXiv:1908.01504}
}
read the original abstract
Successfully tracking the human body is an important perceptual challenge for robots that must work around people. Existing methods fall into two broad categories: geometric tracking and direct pose estimation using machine learning. While recent work has shown direct estimation techniques can be quite powerful, geometric tracking methods using point clouds can provide a very high level of 3D accuracy which is necessary for many robotic applications. However these approaches can have difficulty in clutter when large portions of the subject are occluded. To overcome this limitation, we propose a solution based on fully convolutional neural networks (FCN). We develop an optimized Fast-FCN network architecture for our application which allows us to filter observed point clouds and improve tracking accuracy while maintaining interactive frame rates. We also show that this model can be trained with a limited number of examples and almost no manual labelling by using an existing geometric tracker and data augmentation to automatically generate segmentation maps. We demonstrate the accuracy of our full system by comparing it against an existing geometric tracker, and show significant improvement in these challenging scenarios.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[3]
Dynamic high resolution deformable articulated tracking,
A. Walsman, W. Wan, T. Schmidt, and D. Fox, “Dynamic high resolution deformable articulated tracking,” in 3D Vision (3DV), 2017 International Conference on . IEEE, 2017, pp. 38–47
work page 2017
-
[33]
End-to-end recovery of human shape and pose,
A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik, “End-to-end recovery of human shape and pose,” in Computer Vision and Pattern Regognition (CVPR), 2018
work page 2018
-
[1]
Dart: Dense articulated real-time tracking
T. Schmidt, R. A. Newcombe, and D. Fox, “Dart: Dense articulated real-time tracking.” in Robotics: Science and Systems , vol. 2, no. 1, 2014
work page 2014
-
[2]
M. Ye and R. Yang, “Real-time simultaneous pose and shape es- timation for articulated objects using a single depth camera,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2345–2352
work page 2014
-
[4]
Nonlinear body pose estima- tion from depth images,
D. Grest, J. Woetzel, and R. Koch, “Nonlinear body pose estima- tion from depth images,” in Joint Pattern Recognition Symposium . Springer, 2005, pp. 285–292
work page 2005
-
[5]
Real-time human pose tracking from range data,
V . Ganapathi, C. Plagemann, D. Koller, and S. Thrun, “Real-time human pose tracking from range data,” in European conference on computer vision . Springer, 2012, pp. 738–751
work page 2012
-
[6]
Towards viewpoint invariant 3d human pose estimation,
A. Haque, B. Peng, Z. Luo, A. Alahi, S. Yeung, and L. Fei-Fei, “Towards viewpoint invariant 3d human pose estimation,” inEuropean Conference on Computer Vision . Springer, 2016, pp. 160–177
work page 2016
-
[7]
Vnect: Real-time 3d human pose estimation with a single rgb camera,
D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin, M. Shafiei, H.- P. Seidel, W. Xu, D. Casas, and C. Theobalt, “Vnect: Real-time 3d human pose estimation with a single rgb camera,” ACM Transactions on Graphics (TOG) , vol. 36, no. 4, p. 44, 2017
work page 2017
Show all 35 references
-
[8]
Real-time human pose recognition in parts from single depth images,
J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, “Real-time human pose recognition in parts from single depth images,” in Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on . Ieee, 2011, pp. 1297–1304
2011
-
[9]
Progressive search space reduction for human pose estimation,
V . Ferrari, M. Marin-Jimenez, and A. Zisserman, “Progressive search space reduction for human pose estimation,” in Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on . IEEE, 2008, pp. 1–8
2008
-
[10]
Posetrack: Joint multi-person pose estimation and tracking,
U. Iqbal, A. Milan, and J. Gall, “Posetrack: Joint multi-person pose estimation and tracking,” arXiv preprint arXiv:1611.07727 , 2016
2016 arXiv
-
[11]
Human pose estimation with iterative error feedback,
J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik, “Human pose estimation with iterative error feedback,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 4733–4742
2016
-
[12]
Realtime multi- person 2d pose estimation using part affinity fields,
Z. Cao, T. Simon, S.-E. Wei, and Y . Sheikh, “Realtime multi- person 2d pose estimation using part affinity fields,” arXiv preprint arXiv:1611.08050, 2016
2016 arXiv
-
[13]
Towards accurate multi-person pose estimation in the wild,
G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bre- gler, and K. Murphy, “Towards accurate multi-person pose estimation in the wild,” in CVPR, vol. 3, no. 4, 2017, p. 6
2017
-
[14]
Single-shot multi-person 3d pose estimation from monocular rgb,
D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, S. Sridhar, G. Pons- Moll, and C. Theobalt, “Single-shot multi-person 3d pose estimation from monocular rgb,” in 3D Vision (3DV), 2018 Sixth International Conference on, vol. 3, 2018
2018
-
[15]
Densepose: Dense human pose estimation in the wild,
R. A. G ¨uler, N. Neverova, and I. Kokkinos, “Densepose: Dense human pose estimation in the wild,” arXiv preprint arXiv:1802.00434 , 2018
2018 arXiv
-
[16]
Bodynet: V olumetric inference of 3d human body shapes,
G. Varol, D. Ceylan, B. Russell, J. Yang, E. Yumer, I. Laptev, and C. Schmid, “Bodynet: V olumetric inference of 3d human body shapes,” arXiv preprint arXiv:1804.04875 , 2018
2018 arXiv
-
[17]
Neural body fitting: Unifying deep learning and model-based human pose and shape estimation,
M. Omran, C. Lassner, G. Pons-Moll, P. V . Gehler, and B. Schiele, “Neural body fitting: Unifying deep learning and model-based human pose and shape estimation,” arXiv preprint arXiv:1808.05942 , 2018
2018 arXiv
-
[18]
Real time motion capture using a single time-of-flight camera,
V . Ganapathi, C. Plagemann, D. Koller, and S. Thrun, “Real time motion capture using a single time-of-flight camera,” in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on . IEEE, 2010, pp. 755–762
2010
-
[19]
Personalization and evaluation of a real-time depth- based full body tracker,
T. Helten, A. Baak, G. Bharaj, M. Muller, H.-P. Seidel, and C. Theobalt, “Personalization and evaluation of a real-time depth- based full body tracker,” in 2013 International Conference on 3D Vision (3DV). IEEE, 2013, pp. 279–286
2013
-
[20]
Real-time continuous pose recovery of human hands using convolutional networks,
J. Tompson, M. Stein, Y . Lecun, and K. Perlin, “Real-time continuous pose recovery of human hands using convolutional networks,” ACM Transactions on Graphics (ToG) , vol. 33, no. 5, p. 169, 2014
2014
-
[21]
Learning deconvolution network for semantic segmentation,
H. Noh, S. Hong, and B. Han, “Learning deconvolution network for semantic segmentation,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1520–1528
2015
-
[22]
Fully convolutional networks for semantic segmentation,
J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 3431–3440
2015
-
[23]
U-net: Convolutional networks for biomedical image segmentation,
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” inInternational Confer- ence on Medical image computing and computer-assisted intervention . Springer, 2015, pp. 234–241
2015
-
[24]
Joint multi-person pose estimation and semantic part segmentation
F. Xia, P. Wang, X. Chen, and A. L. Yuille, “Joint multi-person pose estimation and semantic part segmentation.” in CVPR, vol. 2, no. 6, 2017, p. 7
2017
-
[25]
Deep learning for human part discovery in images,
G. L. Oliveira, A. Valada, C. Bollen, W. Burgard, and T. Brox, “Deep learning for human part discovery in images,” in Robotics and Automation (ICRA), 2016 IEEE International Conference on . IEEE, 2016, pp. 1634–1641
2016
-
[26]
Human pose estimation via convolu- tional part heatmap regression,
A. Bulat and G. Tzimiropoulos, “Human pose estimation via convolu- tional part heatmap regression,” in European Conference on Computer Vision. Springer, 2016, pp. 717–732
2016
-
[27]
Deepcontour: A deep convolutional feature learned by positive-sharing loss for contour detection,
W. Shen, X. Wang, Y . Wang, X. Bai, and Z. Zhang, “Deepcontour: A deep convolutional feature learned by positive-sharing loss for contour detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 3982–3991
2015
-
[28]
Tensorflow: a system for large-scale machine learning
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al. , “Tensorflow: a system for large-scale machine learning.” in OSDI, vol. 16, 2016, pp. 265–283
2016
-
[29]
Geometric skinning with approximate dual quaternion blending,
L. Kavan, S. Collins, J. ˇZ´ara, and C. O’Sullivan, “Geometric skinning with approximate dual quaternion blending,” ACM Transactions on Graphics (TOG), vol. 27, no. 4, p. 105, 2008
2008
-
[30]
The xbox one system on a chip and kinect sensor,
J. Sell and O. Patrick, “The xbox one system on a chip and kinect sensor,” IEEE Micro, no. 1, pp. 1–1, 2014
2014
-
[31]
Sun rgb-d: A rgb-d scene un- derstanding benchmark suite,
S. Song, S. P. Lichtenberg, and J. Xiao, “Sun rgb-d: A rgb-d scene un- derstanding benchmark suite,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 567–576
2015
-
[32]
Detect what you can: Detecting and representing objects using holis- tic models and body parts,
X. Chen, R. Mottaghi, X. Liu, S. Fidler, R. Urtasun, and A. Yuille, “Detect what you can: Detecting and representing objects using holis- tic models and body parts,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2014, pp. 1971–1978
2014
-
[34]
Generation of human depth images with body part labels for complex human pose recognition,
K. Nishi and J. Miura, “Generation of human depth images with body part labels for complex human pose recognition,” vol. 71. Elsevier, 2017, pp. 402–413
2017
-
[35]
Very deep convolutional networks for large-scale image recognition,
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
2014 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.