Pith. sign in

REVIEW 4 major objections 4 minor 24 references

THU-Warwick Submission for EPIC-KITCHEN Challenge 2025: Semi-Supervised Video Object Segmentation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A SAM2-initialized visual backbone fused with depth features reaches 90.1% J&F on the EPIC-KITCHENS VISOR test set.

desk verdict A competent but incremental VISOR challenge report; the SAM2-pretraining gain is solid, while the depth contribution is only supported by a single val ablation and should be treated as unverified. read the letter →

arxiv 2506.06748 v1 pith:7BSGNIO2 submitted 2025-06-07 cs.CV

Mingqi Gao , Haoran Duan , Tianlu Zhang , Jungong Han This is my paper
classification cs.CV
keywords Semi-supervisedvideoobjectsegmentationEgocentricvisionEPIC-KITCHENSVISORDepthAnythingV2SAM2CutieMulti-scalefeaturefusionLong-termmemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a semi-supervised video object segmentation system for egocentric video, aimed at the EPIC-KITCHENS VISOR benchmark, where the targets are hands and active objects. The authors argue that visual pretraining from SAM2 and geometric cues from a depth model are complementary: SAM2 supplies strong object perception and frame-to-frame correspondence, while depth helps separate targets from cluttered backgrounds and track through occlusions. They integrate both in the Cutie long-term memory framework, fusing multi-scale Hiera-Large visual features with Depth Anything V2 features through a learnable MLP. The method reaches a J&F of 90.1% on the VISOR test set, with ablations attributing roughly 0.4 J&F on validation to the depth branch and a larger gain to SAM2 initialization.

What carries the argument

The central object is a dual-encoder fusion inside the Cutie video object segmentation framework. The visual encoder is Hiera-Large, a hierarchical vision transformer whose parameters are initialized from SAM2, and the geometric encoder is the Depth Anything V2 network, whose DINOv2-Large backbone and DPT decoder produce multi-scale depth-derived embeddings. Features from the two encoders are extracted at 1/4, 1/8, and 1/16 resolution, resized to handle patch-size mismatches, concatenated scale-wise, and fused by a learnable MLP before entering Cutie's memory readout and mask decoder. This lets the model add geometric context as a trainable side channel while keeping Cutie's object-aware long-term memory.

What would settle it

Re-run the validation ablation with the Depth Anything V2 branch fed scrambled or heavily blurred depth maps while keeping all other settings identical; if J&F stays at 88.7% or higher, the reported depth contribution comes from the fusion module's parameters rather than from geometric information.

Watch

Extended reading notes

Core claim

On the VISOR test set, the segmentation pipeline reaches a J&F score of 90.1% (J 88.1%, F 92.0%) when the Hiera-Large visual encoder is initialized with SAM2 weights, depth features from Depth Anything V2 are fused into the Cutie memory framework, and multi-scale and flip inference are used. On the VISOR validation set, switching the visual backbone from MAE-initialized Hiera-Large to SAM2-initialized Hiera-Large raises J&F from 86.9% to 88.3%, adding the geometric encoder raises it further to 88.7%, and multi-scale-plus-flip post-processing raises it to 88.9%. The paper interprets these results as evidence that large-scale visual pretraining and geometric depth cues jointly improve egocentric video object segmentation, and that long-term memory design still matters because directly fine-tuning SAM2 reaches only 87.8% J&F.

Load-bearing premise

Depth Anything V2 produces depth features that stay accurate and informative in fast-moving, heavily occluded egocentric scenes, and those features add signal beyond what SAM2-pretrained visual features already encode.

Editorial extensions

If this is right

  • If the 90.1% test score holds, combining a SAM2-pretrained visual backbone with depth cues is a strong recipe for egocentric semi-supervised video object segmentation on VISOR.
  • The 1.4-point validation gain from SAM2 initialization over MAE initialization implies that large-scale visual pretraining transfers well to hand-and-object tracking.
  • The roughly 0.4-point depth gain supports adding geometric encoders to memory-based video object segmentation frameworks when scenes are cluttered and occlusions are frequent.
  • Multi-scale and flip fusion add further J&F, so test-time aggregation remains useful even with foundation-model features.
  • Directly fine-tuning SAM2 underperforms the Cutie-based design, indicating that memory architecture still contributes beyond backbone strength.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Depth cues may help most precisely in frames with hand-object occlusions and near-field objects; a per-clip breakdown of the depth ablation on VISOR's interaction-heavy segments would test this.
  • The same dual-encoder fusion could generalize to other egocentric tasks such as active-object detection or affordance segmentation, where depth disambiguates physical contact.
  • Because the fusion module is a small trainable MLP, the architecture leaves room for swapping in other geometric estimators or adding temporal depth consistency as a training signal.
  • If depth features mainly serve as a shortcut for static-scene structure, the depth gain should be larger on background-cluttered clips and near zero on clips where the camera stays still.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents the THU-Warwick submission to the EPIC-KITCHEN VISOR 2025 semi-supervised video object segmentation challenge. The method builds on the Cutie baseline, replaces the visual backbone with a Hiera-Large encoder initialized from SAM2, and adds a geometric encoder from Depth Anything V2 (DINOv2-Large with a DPT decoder) whose multi-scale features are concatenated with visual features and fused by a learnable MLP. Training is two-stage: initialization on the Mega VOS collection with frozen encoders, then fine-tuning on the VISOR train and val sets. The paper reports a J&F of 90.1% on the VISOR test set with multi-scale and flip inference, and ablations on the val set showing gains from SAM2 initialization, depth features, and post-processing.

Significance. If the reported results hold, the paper demonstrates that combining large-scale visual pretraining (SAM2) with geometric depth cues (Depth Anything V2) is beneficial for egocentric video object segmentation, a domain characterized by clutter, occlusion, and rapid viewpoint change. The method is built from publicly available components and the ablation structure is sensible, which makes the findings potentially reproducible and useful for the challenge community. However, the significance is tempered by the absence of error bars or repeated runs, the lack of a test-set ablation that removes the depth stream, and the sparing description of the fusion architecture.

major comments (4)
  1. [4.2 (Table 2)] The only evidence for the depth contribution is a single +0.4 J&F gain on the val set (88.3% without depth versus 88.7% with depth). No error bars, multiple seeds, or per-video breakdowns are reported, and Table 1 never evaluates the test-set configuration without depth. Because a +0.4 gain is exactly the magnitude of the MS+Flip gain on the test set (90.1% versus 89.7%), it is plausible that the depth gain is within run-to-run variance. This is load-bearing because the abstract and introduction attribute the method's success to "depth-based geometric cues." Please provide a test-set ablation without the depth stream, or repeated training runs with variance estimates, to establish that the depth contribution is real and not an artifact of a single run.
  2. [3] The fusion module is underspecified. The text says visual and geometric features are "concatenated at matching scales and subsequently fused via a learnable MLP layer," but it does not state the MLP architecture (depth, hidden width, normalization, activation), whether the three scales are fused independently or jointly, or how the resolution mismatch from different patch sizes is exactly resolved ("we resize the input frames fed into the geometric encoder" is ambiguous regarding the resizing rule). Additionally, the training schedule for the fusion module—whether it is trained in stage 1, stage 2, or both, and whether any encoder weights are unfrozen during VISOR fine-tuning—is not specified. Since the depth-fusion design is the paper's primary novelty, these details are necessary for reproducibility and for assessing whether the reported gain comes from the fusion mechanism or merely from added parameters.
  3. [4.1 (Table 1)] The test-set ablation is incomplete. Both rows in Table 1 include Flip, so there is no condition with neither Flip nor MS, meaning the individual contribution of Flip and MS on the test set cannot be separated. Moreover, the caption of Table 2 introduces "Post" without defining it, and the sentence "The highlighted scores and the bottom results in Table 1 come from the same setting" is confusing: Table 1 reports test-set scores (90.1%) while Table 2 reports val-set scores (88.9%), and the training iterations differ (100k versus 50k). Please define "Post" explicitly and clarify what setting is shared between the two tables.
  4. [4.2] The comparison with "fine-tune the original SAM2 model directly on VISOR" (J&F 87.8%) is used to argue that "effective memory design for complex backgrounds and long-term dynamics is still crucial." However, this comparison conflates several factors: the SAM2 memory mechanism differs from Cutie, the SAM2 baseline does not include the depth stream or the two-stage training, and the number of training iterations may differ. The proper control for isolating SAM2 initialization is the Hiera-L (MAE) row in Table 2, which the paper does include. Please restrict the claim to what the comparison actually supports, or add an ablation that varies only the initialization within the Cutie framework.
minor comments (4)
  1. [Abstract and Introduction] The phrase "aJ&Fscore" appears twice with a missing space; the correct form is "a J&F score."
  2. [3] The notation for feature maps, "{f si v}3 i=1" and "{f si g}3 i=1", is not typeset clearly in the provided text; please use consistent subscripts and superscripts (e.g., f_v^{s_i}) to avoid ambiguity.
  3. [4.1] The term "max skip as 1 when sampling frames as pseudo-training videos" is not defined in this paper; please provide a brief explanation or a reference to the Cutie paper where this hyperparameter is described.
  4. [References] Several references have incomplete or inconsistent bibliographic information, such as [3] and [11] lacking a venue or conference name, and [13] not specifying the venue. Please standardize the reference list.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivational circularity: the reported J&F scores are measured on held-out test and validation sets, and the method's components are external pretrained models plus a trained fusion module.

full rationale

This paper is a challenge report, not a derivation. The central claim is the measured VISOR test-set J&F of 90.1%, which is obtained by training the Cutie baseline with SAM2-initialized Hiera-Large as visual encoder, Depth Anything V2 as geometric encoder, and a learnable MLP fusion module, then evaluating on the held-out test set. There is no equation in which an output is defined in terms of an input, and no parameter is fitted to the test set and then reported as a prediction. The depth ablation in Table 2 compares a single run without depth (88.3 J&F) to a single run with depth (88.7 J&F, 88.9 after post-processing) on the validation set; this is a limited empirical comparison with no error bars, but it is not circular because the depth stream is not constructed from the validation labels or from the test score. The use of SAM2, Cutie, and Depth Anything V2 is justified by external pretrained models and prior benchmark results, not by self-citation; the only self-citation is the authors' own survey [8], which is not load-bearing for any experimental claim. Multi-scale and flip inference selection after observing validation behavior is normal benchmark practice and is a generalization-risk concern, not circularity. Therefore, no circular step can be quoted or reduced, and the appropriate score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central result rests on pretrained model transfer, benchmark reliability, and a set of hand-chosen hyperparameters. No new theoretical entities or mechanisms are introduced. The most important free parameters are the learned fusion and decoder weights and the VISOR fine-tuning settings, all fitted to data or chosen without a reported sweep.

free parameters (6)
  • Fusion MLP weights = learned during training
    The MLP fuses visual and depth features at three scales and is trained on Mega VOS data and VISOR train/val. These learned weights directly affect the reported score.
  • Cutie decoder and memory module weights = learned during training
    The baseline modules are trained on Mega VOS data and then fine-tuned on VISOR. The reported J&F depends entirely on this training.
  • VISOR fine-tuning schedule = lr 5e-5, weight decay 0.5, batch 8, 100k iterations
    Hand-chosen hyperparameters reported in Section 4.1 with no sweep or sensitivity analysis.
  • Max skip for pseudo-training video sampling = 1
    Chosen because VISOR annotations are sparse and long-term context is needed, as stated in Section 4.1.
  • Multi-scale inference factors = 1.2x, 1.3x, 1.4x
    Selected by hand for test-time fusion; combined with flip, this contributes a 0.2 J&F gain on the validation set.
  • Horizontal flip inference = enabled
    Test-time augmentation chosen for the final setting in Tables 1 and 2.
assumptions (4)
  • domain assumption SAM2-pretrained Hiera-Large features generalize to egocentric VISOR objects and scenes.
    The method bets on transfer from large-scale visual pretraining. Section 4.2 ablates the effect but does not independently validate why the transfer works.
  • domain assumption Depth Anything V2 produces depth features accurate enough for egocentric, heavily occluded scenes.
    Depth is added in Section 3 with no analysis of depth quality in egocentric settings.
  • domain assumption VISOR benchmark annotations and the J&F evaluation protocol are trustworthy.
    All reported scores depend on the benchmark protocol introduced by Darkhalil et al. [5].
  • domain assumption Cutie is a sound baseline and replacing only the backbone does not require other architectural changes.
    The paper adopts Cutie [4] as baseline and keeps other modules unchanged, as stated in Section 3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of THU-Warwick Submission for EPIC-KITCHEN Challenge 2025: Semi-Supervised Video Object Segmentation." pith.science (2026). https://pith.science/paper/7BSGNIO2

@misc{pith2026250606748,
  author       = {Pith},
  title        = {Pith review of: THU-Warwick Submission for EPIC-KITCHEN Challenge 2025: Semi-Supervised Video Object Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7BSGNIO2}},
  note         = {Machine review of arXiv:2506.06748}
}
read the original abstract

In this report, we describe our approach to egocentric video object segmentation. Our method combines large-scale visual pretraining from SAM2 with depth-based geometric cues to handle complex scenes and long-term tracking. By integrating these signals in a unified framework, we achieve strong segmentation performance. On the VISOR test set, our method reaches a J&F score of 90.1%.

Figures

Figures reproduced from arXiv: 2506.06748 by the authors.

Figure 1
Figure 1. Overview of our solution. The success of SAM2 [18] highlights that large-scale training can significantly improve segmentation perfor￾mance, even with simple memory strategies that consider only the first and most recent frames. However, such limited memory usage is less effective for long-term videos. As a result, many follow-up works [7, 20, 23] have explored bet￾ter memory selection and utilisation to boost perfo… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 16 canonical work pages

  1. [1]

    One- shot video object segmentation

    Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taix´e, Daniel Cremers, and Luc Van Gool. One- shot video object segmentation. InCVPR, pages 221–230,

    work page

  2. [2]

    Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model

    Ho Kei Cheng and Alexander G Schwing. Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model. InECCV, pages 640–658. Springer, 2022. 1

    work page 2022

  3. [3]

    Rethink- ing space-time networks with improved memory coverage for efficient video object segmentation

    Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Rethink- ing space-time networks with improved memory coverage for efficient video object segmentation. 34:11781–11794,

    work page

  4. [4]

    Putting the object back into video object segmentation

    Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. InCVPR, 2024. 1, 2

    work page 2024

  5. [5]

    Epic-kitchens visor benchmark: Video segmenta- tions and object relations

    Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmenta- tions and object relations. InProceedings of the Neural In- formation Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2022. 1

    work page 2022

  6. [6]

    MOSE: A new dataset for video object segmentation in complex scenes

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. MOSE: A new dataset for video object segmentation in complex scenes. InICCV,

    work page

  7. [7]

    Sam2long: Enhancing sam 2 for long video seg- mentation with a training-free memory tree.arXiv preprint arXiv:2410.16268, 2024

    Shuangrui Ding, Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Yuwei Guo, Dahua Lin, and Jiaqi Wang. Sam2long: Enhancing sam 2 for long video seg- mentation with a training-free memory tree.arXiv preprint arXiv:2410.16268, 2024. 2

    arXiv

  8. [8]

    Deep learning for video object segmentation: a review.Artificial Intelligence Review, 56(1):457–531, 2023

    Mingqi Gao, Feng Zheng, James JQ Yu, Caifeng Shan, Guiguang Ding, and Jungong Han. Deep learning for video object segmentation: a review.Artificial Intelligence Review, 56(1):457–531, 2023. 1

    work page 2023

Show all 24 references
  1. [9]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. InCVPR, pages ...

    2024

  2. [10]

    Lvos: A benchmark for large- scale long-term video object segmentation.arXiv preprint arXiv:2404.19326, 2024

    Lingyi Hong, Zhongying Liu, Wenchao Chen, Chenzhi Tan, Yuang Feng, Xinyu Zhou, Pinxue Guo, Jinglun Li, Zhaoyu Chen, Shuyong Gao, et al. Lvos: A benchmark for large- scale long-term video object segmentation.arXiv preprint arXiv:2404.19326, 2024. 1

    2024 arXiv

  3. [11]

    Video object segmentation with adaptive feature bank and uncertain-region refinement

    Yongqing Liang, Xin Li, Navid Jafari, and Jim Chen. Video object segmentation with adaptive feature bank and uncertain-region refinement. 33:3430–3441, 2020. 1

    2020

  4. [12]

    Video object segmentation using space-time memory networks

    Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. InICCV, pages 9226–9235, 2019. 1

    2019

  5. [13]

    Maxime Oquab, Timoth ´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Rus- sell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Ni...

    2023

  6. [14]

    Hd-epic: A highly-detailed egocentric video dataset

    Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: A highly-detailed egocentric video dataset. InCVPR, 2025. 1

    2025

  7. [15]

    An outlook into the fu- ture of egocentric vision.IJCV, 132(11):4880–4936, 2024

    Chiara Plizzari, Gabriele Goletto, Antonino Furnari, Sid- dhant Bansal, Francesco Ragusa, Giovanni Maria Farinella, Dima Damen, and Tatiana Tommasi. An outlook into the fu- ture of egocentric vision.IJCV, 132(11):4880–4936, 2024. 1

    2024

  8. [16]

    The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017. 1

    2017 arXiv

  9. [17]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. InICCV, pages 12179–12188, 2021. 2

    2021

  10. [18]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. 1, 2

    2024 arXiv

  11. [19]

    Hi- era: A hierarchical vision transformer without the bells-and- whistles

    Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al. Hi- era: A hierarchical vision transformer without the bells-and- whistles. InICML, pages 29441–29454. PMLR, 2023. 2

    2023

  12. [20]

    A distractor-aware memory for visual object tracking with sam2.arXiv preprint arXiv:2411.17576, 2024

    Jovana Videnovic, Alan Lukezic, and Matej Kristan. A distractor-aware memory for visual object tracking with sam2.arXiv preprint arXiv:2411.17576, 2024. 2

    2024 arXiv

  13. [21]

    Feelvos: Fast end-to-end embedding learning for video object segmenta- tion

    Paul V oigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, and Liang-Chieh Chen. Feelvos: Fast end-to-end embedding learning for video object segmenta- tion. InCVPR, pages 9481–9490, 2019. 1

    2019

  14. [22]

    Youtube-vos: A large-scale video object segmentation benchmark.arXiv preprint arXiv:1809.03327, 2018

    Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark.arXiv preprint arXiv:1809.03327, 2018. 1

    2018 arXiv

  15. [23]

    Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory.arXiv preprint arXiv:2411.11922,

    Cheng-Yen Yang, Hsiang-Wei Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory.arXiv preprint arXiv:2411.11922,

    arXiv

  16. [24]

    Depth any- thing v2.arXiv:2406.09414, 2024

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2.arXiv:2406.09414, 2024. 1, 2 4

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.