Pith. sign in

REVIEW 4 major objections 4 minor 25 references

In defense of OSVOS

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding video loss lifts OSVOS from 75.0 to 76.2 J-mean on DAVIS-2016, and to 67.5 with MobileNet.

desk verdict Reasonable auxiliary-loss idea, but the central comparison is against an under-trained OSVOS baseline, so the headline claim doesn't survive. read the letter →

arxiv 1908.06692 v2 pith:FI2WEI6C submitted 2019-08-19 cs.CV

Yu Liu , Yutong Dai , Anh-Dzung Doan , Lingqiao Liu , Ian Reid This is my paper · ORCID
classification cs.CV
keywords videoobjectsegmentationOSVOSlossinstance-awarelearningonlinefine-tuningDAVIS-2016tripletcontrastivecenter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the parent-network training phase of one-shot video object segmentation (OSVOS) overemphasizes generic objectness and washes out instance-level cues, which slows the later online fine-tuning step. To fix this, it adds a lightweight video-loss head that supervises the parent network with video identity during training on DAVIS-2016. The paper reports that this module raises the final J-mean on DAVIS-2016 from 75.0 to 76.2 with VGG16 and from 65.7 to 67.5 with MobileNet when the mixed instance-aware video loss is used. If correct, the result matters because video loss is a small, common building block that can be attached to any fine-tuning-based video object segmentation method without changing inference cost.

What carries the argument

The mechanism is the video loss block: a lightweight auxiliary head attached in parallel with the prediction head during parent-network training, supervised by the identity of the training video rather than by the binary foreground/background label alone. It comes in three forms: 2D-VL applies the same weighted cross-entropy expression to per-video categories in image coordinates; HD-VL maps features into a high-dimensional embedding and applies a triplet loss over sampled foreground/background points; and the mixed loss adds a contrastive center loss that penalizes the distance between foreground and background cluster centers. The block works by enforcing that frames of the same video map near one another while different videos map far apart, an 'average object' per sequence, so the parent network keeps instance cues that online fine-tuning can exploit.

What would settle it

Run both methods to 20k online fine-tuning iterations on DAVIS-2016 and compare J-mean; if the video-loss gain vanishes or reverses at that budget, the reported improvement is an artifact of the shortened training protocol.

Watch

Extended reading notes

Core claim

The central claim is that adding video-level constraints to step (2) of OSVOS, training the parent network, produces a parent representation that preserves instance identity, so that step (3), online fine-tuning on the first frame of a test video, adapts faster and more accurately. The paper formalizes the constraint through three losses: a two-dimensional video loss that treats each training video as a category in image space; a high-dimensional triplet loss that pulls foreground pixels together and foreground away from background in an embedding space; and a mixed instance-aware video loss combining triplet loss with a contrastive center loss that keeps the foreground and background cluster centers apart. On DAVIS-2016, OSVOS with the two-dimensional video loss reaches 76.2 J-mean with VGG16 and 66.2 with MobileNet, and the mixed loss reaches 67.5 with MobileNet, all measured after 10k online fine-tuning iterations without post-processing. The authors read this as evidence that instance-level supervision in the parent phase, not any change to the fine-tuning phase, is what removes the noisy, similar-looking false positives that limit OSVOS.

Load-bearing premise

The claim rests on comparing OSVOS and OSVOS-VL at 10k online fine-tuning iterations; if the improvement disappears when both are run to the 20k iterations needed for OSVOS to reach its originally reported scores, the central claim fails.

Editorial extensions

If this is right

  • On DAVIS-2016, adding the video loss raises the online fine-tuning J-mean from 75.0 to 76.2 with a VGG16 backbone and from 65.7 to 67.5 with MobileNet when the mixed loss is used.
  • The gains are concentrated in sequences with abrupt motion or distractors that resemble the target object, such as bmx-trees, dog, drift-chicane, and paragliding-launch.
  • Because video loss is a common module, it should transfer to other fine-tuning-based video object segmentation methods such as OSVOS-S, OnAVOS, and CINM, and to tasks like depth estimation and saliency detection.
  • The extra training cost is small: the video-loss head runs in parallel with prediction and does not change the online fine-tuning procedure or inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper compares all methods at 10k online fine-tuning iterations and states that the OSVOS baseline reaches the original reported numbers only around 20k iterations; no video-loss results are reported at 20k, so whether the advantage persists at the canonical budget is untested.
  • If the mechanism is instance separability in the parent embedding, then the same recipe, discriminative instance-level pretraining before few-shot adaptation, should transfer to other few-shot settings where a base model is fine-tuned to a single exemplar, such as few-shot classification or tracking.
  • A direct way to test the proposed mechanism would be to measure nearest-neighbor distances in the parent network's embedding: sequences where the target instance is close to a distractor should show the largest improvements, and the per-sequence table already hints at that pattern.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a 'video loss' module to be added during the parent-network training stage of OSVOS, a fine-tuning-based video object segmentation method. Three variants are introduced: a two-dimensional video loss that uses video identity to separate object instances, a high-dimensional triplet loss, and a mixed instance-aware loss combining contrastive center loss and triplet loss. Experiments on DAVIS-2016 with VGG16 and MobileNet backbones report J-mean improvements after online fine-tuning over the OSVOS baseline (e.g., 76.2 vs. 75.0 for VGG16, 67.5 vs. 65.7 for MobileNet). The paper also claims the module is a common building block generalizable to other fine-tuning methods and to tasks such as depth estimation and saliency detection.

Significance. If the claim holds, the contribution is a simple, lightweight loss that improves instance discrimination during parent-network training, potentially benefiting any fine-tuning-based method. The experimental evidence, however, is weakened by the evaluation protocol: all fine-tuning results are reported at 10k iterations, while the paper itself notes that OSVOS replicates the published numbers only around 20k iterations. No video-loss results are provided at that standard setting, so the claimed gains may reflect faster early convergence rather than a higher final accuracy. Furthermore, the paper does not report several loss hyperparameters, making the method hard to reproduce, and the per-sequence tables show substantial regressions on some videos with single-run results. These issues are addressable with additional experiments and reporting, but they currently prevent the central claim from being considered established.

major comments (4)
  1. [Section 4.2, Tables 1-2] All fine-tuning results are reported after 10k online iterations, while the paper states in Section 4.2 that around 20k iterations the OSVOS baseline replicates the numbers originally reported in [2]. No OSVOS-VL variants are reported at the 20k setting, so the claimed improvements (76.2 vs. 75.0 for VGG16, 67.5 vs. 65.7 for MobileNet) are established only against an under-trained OSVOS baseline. The central claim of the abstract requires evidence at the standard OSVOS configuration; please provide J-mean at 20k iterations for all proposed variants and baselines.
  2. [Sections 3.3-3.4, Eqs. (2)-(4)] The hyperparameters beta1, beta2, and lambda (the margin in the triplet and center losses), as well as the embedding dimension and the number of sampled points per part, are never reported. These are free parameters of the proposed losses, and without them the mixed loss in Eq. (4) is not reproducible. The paper should specify these values and, ideally, provide a sensitivity analysis to show that the reported gains are not tied to a particular tuning.
  3. [Section 4.4, Table 3] Per-sequence results show large degradations for several sequences (e.g., car-roundabout: 88.3 to 75.6; scooter-black: 58.2 to 35.4 on VGG16). All experiments appear to be single runs, and the paper reports no error bars or multiple seeds. Given these large per-sequence regressions, the average gain of about 1.2 J-mean could easily lie within run-to-run variance. The claim of 'significant improvement' requires either multiple runs or statistical testing.
  4. [Abstract, Section 2.4, and Conclusion] The paper repeatedly claims that video loss generalizes to other fine-tuning based methods (e.g., OnVOS, OSVOS-S, CINM) and to similar vision tasks such as depth estimation and saliency detection, but no experiments support these claims. Since this is part of the paper's stated contribution, either provide supporting experiments for at least one other method or task, or explicitly restrict the claims to OSVOS on DAVIS-2016.
minor comments (4)
  1. [Throughout] There are several typos and misspellings, including 'DAVIS' appearing as 'DA VIS' in the abstract and Section 1, 'onlie' for 'online', 'sourrondings' for 'surroundings', and 'dimensioins' for 'dimension'. The paper would benefit from a careful proofreading pass.
  2. [Section 3.2] The implementation of 2D-VL is unclear: 'we add the identity of each video (vid) into the training process as input' and 'our 2D-VL only updates corresponding (video) category directly' do not specify how the additional head is structured, how the video identity is encoded, or how gradients are routed. Please provide a precise description of the 2D-VL head and its loss computation.
  3. [Section 3.5] The sentence 'For training the parent network, fine-tuning of 240 epochs is conducted based on the initialization of ImageNet features' is ambiguous because the phrase 'fine-tuning' is also used for online fine-tuning later in the same paragraph. Clarify which stage uses 240 epochs and which uses 10k iterations.
  4. [Figure 4] The caption of Figure 4 does not explain the meaning of the dashed arrows or the 'Pull force' and 'Push force' annotations. Expanding the caption would help the reader understand the proposed high-dimensional video loss.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical comparison against the external OSVOS baseline and the DAVIS-2016 benchmark, with no load-bearing self-reference or constructional equivalence.

full rationale

The paper's central claim is that adding a video-loss module during parent-network training improves the final online-fine-tuned segmentation accuracy compared with OSVOS. This is an empirical claim tested against an external baseline and external benchmark; it is not derived from the loss definitions by construction. The video losses are explicitly formulated as weighted cross-entropy, triplet, contrastive-center, and a weighted mixture of these, and the paper reports their measured J-mean values. The 2D video loss admittedly 'share[s] the same expression with Equation 1' but is applied to a different supervision target, namely the video identity, so this is an architectural/training choice rather than a case of predicting a fitted quantity. The paper's own note that all experiments use 10k iterations and that the OSVOS baseline 'around 20k iterations' replicates the original reported numbers is a legitimate experimental-validity concern about whether the comparison uses an under-trained baseline, but it is not circularity: the comparison is still against an external, independently published system. No load-bearing uniqueness theorem, self-citation chain, or fitted-input-called-prediction step appears. The paper is self-contained in the sense required: its evidence is externally observable performance on DAVIS-2016, and no central result reduces by definition to its own inputs.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim depends on a handful of unreported hyperparameters and on several assumptions about feature-space geometry and transfer of video-id supervision. No fundamentally new entity is introduced.

free parameters (6)
  • beta1 (weight of contrastive center loss)
    Coefficient in Eq. 4 balancing Lhd_cl and Lhd_tl; value not reported.
  • beta2 (weight of triplet loss)
    Coefficient in Eq. 4; value not reported.
  • lambda (margin for triplet and center losses)
    Margin in Eqs. 2 and 3; value not reported.
  • embedding dimension = 20
    Table 2 caption states 20 dimensions for OSVOS-Vhd and OSVOS-Vmixed embeddings.
  • sampled points per part = 256
    Section 3.3 says 256 points are randomly sampled in foreground and background parts.
  • online fine-tuning iterations = 10k
    Section 4.2 states all experiments use 10k iterations; this protocol choice is central to the comparison.
assumptions (3)
  • domain assumption Different objects are linearly separable in high-dimensional feature space, and intra-object feature distances are smaller than inter-object distances.
    Invoked in Sec. 3.1 to justify the video loss design.
  • ad hoc to paper Video identity of training frames is a useful proxy for object-instance identity and its learned features transfer to unseen test videos.
    The 2D-VL (Sec. 3.2) supervises a video-id classification head, assuming this supervision generalizes to novel objects at test time.
  • domain assumption OSVOS's three-step training framework (ImageNet features, parent network on DAVIS, online fine-tuning) is an appropriate base; the paper introduces no alternative to this framework.
    The entire method is built on OSVOS and assumes the framework's validity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of In defense of OSVOS." pith.science (2026). https://pith.science/paper/FI2WEI6C

@misc{pith2026190806692,
  author       = {Pith},
  title        = {Pith review of: In defense of OSVOS},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FI2WEI6C}},
  note         = {Machine review of arXiv:1908.06692}
}
read the original abstract

As a milestone for video object segmentation, one-shot video object segmentation (OSVOS) has achieved a large margin compared to the conventional optical-flow based methods regarding to the segmentation accuracy. Its excellent performance mainly benefit from the three-step training mechanism, that are: (1) acquiring object features on the base dataset (i.e. ImageNet), (2) training the parent network on the training set of the target dataset (i.e. DAVIS-2016) to be capable of differentiating the object of interest from the background. (3) online fine-tuning the interested object on the first frame of the target test set to overfit its appearance, then the model can be utilized to segment the same object in the rest frames of that video. In this paper, we argue that for the step (2), OSVOS has the limitation to 'overemphasize' the generic semantic object information while 'dilute' the instance cues of the object(s), which largely block the whole training process. Through adding a common module, video loss, which we formulate with various forms of constraints (including weighted BCE loss, high-dimensional triplet loss, as well as a novel mixed instance-aware video loss), to train the parent network in the step (2), the network is then better prepared for the step (3), i.e. online fine-tuning on the target instance. Through extensive experiments using different network structures as the backbone, we show that the proposed video loss module can improve the segmentation performance significantly, compared to that of OSVOS. Meanwhile, since video loss is a common module, it can be generalized to other fine-tuning based methods and similar vision tasks such as depth estimation and saliency detection.

Figures

Figures reproduced from arXiv: 1908.06692 by the authors.

Figure 1
Figure 1. A visualized example of OSVOS and OSVOS-VL. (a) Image (b) Ground truth (c) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The workflow of OSVOS-VL. Compared to OSVOS, only a tiny head, video loss [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The illustration of two-dimensional video loss. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The illustration of high-dimensional video loss. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Visualization results of OSVOS and OSVOS-VL. (a) Input (b) Ground Truth (c) [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 17 canonical work pages

  1. [2]

    One-shot video object segmentation

    Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cre- mers, and Luc Van Gool. One-shot video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 221–230, 2017

    work page 2017

  2. [1]

    Cnn in mrf: Video object segmentation via inference in a cnn-based higher-order spatio-temporal mrf

    Linchao Bao, Baoyuan Wu, and Wei Liu. Cnn in mrf: Video object segmentation via inference in a cnn-based higher-order spatio-temporal mrf. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5977–5986, 2018

    work page 2018

  3. [3]

    Openpose: realtime multi-person 2d pose estimation using part affinity fields

    Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: realtime multi-person 2d pose estimation using part affinity fields. arXiv preprint arXiv:1812.08008, 2018

    arXiv 2018

  4. [4]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and ma- chine intelligence, 40(4):834–848, 2018

    work page 2018

  5. [5]

    Blazingly fast video object segmentation with pixel-wise metric learning

    Yuhua Chen, Jordi Pont-Tuset, Alberto Montes, and Luc Van Gool. Blazingly fast video object segmentation with pixel-wise metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1189–1198, 2018

    work page 2018

  6. [6]

    Segflow: Joint learning for video object segmentation and optical flow

    Jingchun Cheng, Yi-Hsuan Tsai, Shengjin Wang, and Ming-Hsuan Yang. Segflow: Joint learning for video object segmentation and optical flow. In Proceedings of the IEEE international conference on computer vision, pages 686–695, 2017

    work page 2017

  7. [7]

    Fast and accurate online video object segmentation via tracking parts

    Jingchun Cheng, Yi-Hsuan Tsai, Wei-Chih Hung, Shengjin Wang, and Ming-Hsuan Yang. Fast and accurate online video object segmentation via tracking parts. In Pro- VIDEO SEGMENTA TION: IN DEFENSE OF OSVOS 11 ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7415–7424, 2018

    work page 2018

  8. [8]

    Semantic instance segmentation with a discriminative loss function

    Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with a discriminative loss function. arXiv preprint arXiv:1708.02551, 2017

    arXiv 2017

Show all 25 references
  1. [9]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  2. [10]

    Fast r-cnn

    Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015

    2015

  3. [11]

    Mobilenets: Effi- cient convolutional neural networks for mobile vision applications

    Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Effi- cient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017

    2017 arXiv

  4. [12]

    Path aggregation network for instance segmentation

    Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8759–8768, 2018

    2018

  5. [13]

    Learning pairwise relationship for multi-object detection in crowded scenes

    Yu Liu, Lingqiao Liu, Hamid Rezatofighi, Thanh-Toan Do, Qinfeng Shi, and Ian Reid. Learning pairwise relationship for multi-object detection in crowded scenes. arXiv preprint arXiv:1901.03796, 2019

    1901 arXiv

  6. [14]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015

    2015

  7. [15]

    Premvos: Proposal- generation, refinement and merging for video object segmentation

    Jonathon Luiten, Paul V oigtlaender, and Bastian Leibe. Premvos: Proposal- generation, refinement and merging for video object segmentation. arXiv preprint arXiv:1807.09190, 2018

    2018 arXiv

  8. [16]

    Video object segmentation without temporal in- formation

    Kevis-Kokitsi Maninis, Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. Video object segmentation without temporal in- formation. arXiv preprint arXiv:1709.06031, 2017

    2017 arXiv

  9. [17]

    A benchmark dataset and evaluation methodology for video object segmentation

    Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    2016

  10. [18]

    Learning video object segmentation from static images

    Federico Perazzi, Anna Khoreva, Rodrigo Benenson, Bernt Schiele, and Alexander Sorkine-Hornung. Learning video object segmentation from static images. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2663–2672, 2017

    2017

  11. [19]

    Yolov3: An incremental improvement

    Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018

    2018 arXiv

  12. [20]

    Mask propagation network for video object segmentation

    Jia Sun, Dongdong Yu, Yinghong Li, and Changhu Wang. Mask propagation network for video object segmentation. arXiv preprint arXiv:1810.10289, 2018. 12 VIDEO SEGMENTA TION: IN DEFENSE OF OSVOS

    2018 arXiv

  13. [21]

    Online adaptation of convolutional neural net- works for video object segmentation

    Paul V oigtlaender and Bastian Leibe. Online adaptation of convolutional neural net- works for video object segmentation. arXiv preprint arXiv:1706.09364, 2017

    2017 arXiv

  14. [22]

    Fast online object tracking and segmentation: A unifying approach

    Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr. Fast online object tracking and segmentation: A unifying approach. arXiv preprint arXiv:1812.05050, 2018

    2018 arXiv

  15. [23]

    Fast video object segmentation by reference-guided mask propagation

    Seoung Wug Oh, Joon-Young Lee, Kalyan Sunkavalli, and Seon Joo Kim. Fast video object segmentation by reference-guided mask propagation. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7376–7385, 2018

    2018

  16. [24]

    Holistically-nested edge detection

    Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE international conference on computer vision, pages 1395–1403, 2015

    2015

  17. [25]

    Efficient video object segmentation via network modulation

    Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang, and Aggelos K Katsagge- los. Efficient video object segmentation via network modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6499–6507, 2018

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.