{"id":"4d8cbd42-bcad-4929-8bb5-f099d1a3aa5d","arxiv_id":"2412.01471","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new large-scale video dataset with dense masks for objects, parts, and backgrounds, plus a memory-based SAM video segmentation model that wins on that benchmark.","lead":"The authors introduce MUG-VOS, a large video dataset whose masks cover obvious objects, background regions, and partial objects, not just the main foreground. They also present a memory-based model that tracks these varied masks better than existing methods on the new benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset validity hinges on SAM-pipeline coverage; test set only refines SAM-proposed tracks, so objects SAM misses are absent from the benchmark and MMPM's SAM-based advantage may be baked in.","rationale":"The reader's weakest-assumption analysis identified exactly the same load-bearing risk: the SAM-based pipeline defines what counts as ground truth, and the test set only refines tracks that SAM already proposed. This is the most serious issue because it undermines both the dataset contribution and the MMPM ranking claim. A dataset whose object coverage is bounded by SAM's proposal recall cannot be claimed to measure multi-granularity video object segmentation in general; it measures performance on SAM-like granularities. The MMPM model, using a SAM encoder, is therefore at an advantage for reasons that are architectural, not necessarily indicative of better multi-granularity tracking. The paper does not provide any independent validation of the pipeline's completeness (e.g., comparison to from-scratch human annotations or alternative proposal methods). The concrete test I propose would directly measure whether the SAM-proposal bottleneck exists by comparing the test set against independent human annotations on a sample of the same videos. If the test passes (high recall and high IoU), the concern is resolved; if it fails, the dataset's core claim is weakened and the MMPM result would have to be reinterpreted. I do not see other concerns as more load-bearing: the Table 2 caption error is a clear presentation mistake but does not change the central validity question; the absence of variance/significance on a 30-video test is important but secondary; the lack of SAM 2 is a completeness gap, not a logical flaw. Therefore the reader's CONDITIONAL verdict is appropriate, and my read does not change it.","tokens_in":14357,"tokens_out":2836,"duration_ms":26223,"concrete_test":"Select a stratified sample of 10–20 videos from the MUG-VOS test set (all drawn from DAVIS-2017 validation). Have independent annotators, without access to SAM-generated proposals or the MUG-VOS tracks, enumerate and segment every object/part/stuff they can identify in the first frame and then track those masks through the video using a different tool (e.g., manual contour tracking or point tracking plus manual correction). Compare the resulting tracks to the MUG-VOS test tracks: compute (a) per-video track-count recall (fraction of human-enumerated objects that have a matching MUG-VOS track), (b) J&F on matched tracks, and (c) track identity consistency across occlusions. If track-count recall is substantially below 1 or matched-track J&F is below ~0.9, then the SAM-proposal bottleneck is real and the benchmark's coverage and correctness claims are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MUG-VOS is a large-scale, densely annotated multi-granularity video segmentation dataset and that MMPM achieves the best performance on it. This claim rests on the assumption that the SAM-plus-optical-flow pipeline produces complete and correct mask tracks. That assumption is least secure for two distinct reasons. First, coverage is bounded by SAM's proposal recall: Algorithm 1 initializes masks from grid point prompts on the first frame, and the forward pass selects among SAM's candidate masks. If SAM does not propose an object (small, non-salient, heavily occluded, or unusual granularity), no track is created for it, and the test-set quality assurance protocol does not fix this because annotators only accept, reject, or refine SAM-proposed tracks, not discover missing objects from scratch. The paper's own appendix says supervisors 'select multi-granularity masks from the first frame using SAM,' confirming that human curation is constrained to SAM's mask vocabulary. Second, the temporal linking uses flow warping and IoU matching, which can lose identity across occlusion, motion blur, or large deformation; the resulting track errors are inherited by both training and test sets. The training set (77,994 clips, 47M masks) receives no human verification at all, while the test set (30 DAVIS videos) is human-refined but still starts from the same pipeline. Since MMPM uses a pretrained SAM encoder, it is evaluated on a distribution strongly shaped by SAM's outputs, so its reported 86.1 J&F versus 85.6 for DEVA and 83.0 for XMem may reflect distributional familiarity rather than general multi-granularity tracking ability. The benchmark's validity as a measurement of 'multi-granularity video object segmentation' therefore rests on an unverified equivalence between SAM's proposal distribution and the true distribution of segmentable objects in video.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MUG-VOS, a video object segmentation dataset with multi-granularity mask annotations obtained by combining SAM with optical-flow-based tracking, and proposes MMPM, a memory-enhanced model built on a pretrained SAM encoder. The authors report 77,994 training video clips with 47M masks, a human-refined test set on 30 DAVIS-2017 videos, and quantitative results (Table 2) in which MMPM achieves 86.1 J&F, outperforming DEVA (85.6) and XMem (83.0). The paper also includes ablations on memory filtering, memory types, update interval, and memory size, plus a DAVIS-2017 evaluation in Appendix Table A.1.","tokens_in":14568,"tokens_out":4635,"duration_ms":38649,"significance":"If the dataset is valid, it addresses a genuine gap: existing VOS benchmarks restrict annotations to salient objects, while MUG-VOS aims to cover non-salient objects, parts, and stuff. The scale is substantial, and the semi-automatic annotation pipeline is a practical contribution. The release of the dataset, code, and a public project page would be valuable to the community. However, the validity of the benchmark and the strength of the model claim are not yet established because the annotation pipeline may miss objects entirely and the experimental evaluation has significant methodological issues.","major_comments":[{"comment":"The caption of Table 2 states that the evaluation is on the DAVIS-2017 validation set, while the running text says 'Table 2 shows the quantitative results on MUG-VOS dataset' and the table header itself reads 'MUG-VOS Test.' Appendix Table A.1 reports a separate DAVIS-2017 evaluation with markedly different scores (e.g., MMPM J&F = 69.1 in Table A.1 versus 86.1 in Table 2). This mislabeling makes it impossible to verify the central claim that MMPM achieves state-of-the-art performance on MUG-VOS, which is the paper's main result. The authors must correct the caption and clearly separate the MUG-VOS test results from the DAVIS results.","section":"Experiments, Table 2"},{"comment":"All ablations are carried out on the MUG-VOS test set, and the final configuration (memory filtering rule 'P. first & last', memory update interval r=5, and N=10 temporal memory values) is selected based on test-set performance. The same test set is then used to report the headline comparison in Table 2. This creates an optimism bias: the reported 86.1 J&F is an estimate of performance after tuning on the evaluation set, not a fair estimate of performance on new videos. A held-out validation split, or cross-validation, is required before the claim of superior performance can be supported.","section":"Ablation study, Tables 3-6"},{"comment":"The ground-truth mask tracks in the MUG-VOS test set are produced by a pipeline in which annotators only accept, reject, or refine SAM-proposed masks; supervisors initially select first-frame masks using SAM. Consequently, any object or granularity that SAM does not propose is entirely absent from the benchmark. Since MMPM uses a frozen SAM encoder, its evaluation is biased toward the vocabulary of masks SAM can produce, which may inflate its apparent advantage over methods not built on SAM. The paper should provide evidence of recall with respect to exhaustive human annotation, for example by having annotators draw masks from scratch on a subset of videos and measuring how many pipeline-generated tracks are missed.","section":"Dataset, Quality assurance and Appendix A"},{"comment":"The reported improvement of MMPM over DEVA is 0.5 J&F points (86.1 versus 85.6) on a test set of only 30 videos. The paper does not report standard deviations, multiple runs, or any significance test. Given the small test set and the fact that hyperparameters were selected on the same set, the evidence is not sufficient to conclude that MMPM 'leads to the best performance among the existing video object segmentation methods.'","section":"Experiments, Table 2"}],"minor_comments":[{"comment":"The 'MUG-VOS Train' row reports 'Annotated Frames 77,9940,' which appears to contain a typo; the number is also inconsistent with 77,994 clips and 47M masks under the given masks-per-frame density (47M / 66.3 ≈ 709k frames). Please clarify the correct number of frames.","section":"Table 1"},{"comment":"The notation 'arg max_{C^i_t} IoU' conflates the candidate-mask set and the track index i; the argument of the maximization is the set of candidate masks for a fixed track, so the expression should be written as 'M^i_t = arg max_{m in C^i_t} IoU(...)' for clarity.","section":"Data collection pipeline, Eq. (6)"},{"comment":"The phrase 'Segment SAM-based video segmentation methods' is awkward; consider 'SAM-based video segmentation methods.'","section":"Abstract and Introduction"},{"comment":"The dataset name DAVIS is consistently typeset as 'DA VIS' with a space; this should be fixed throughout the manuscript, including in the captions and references.","section":"Throughout"},{"comment":"The captions for Figures A.9 and A.10 appear to be copied from Appendix D, as they describe comparisons of the data collection pipeline with DAVIS/Youtube-VOS/UVO, whereas the figures and surrounding text describe qualitative comparisons of MMPM with other methods. The captions should match the content of the figures.","section":"Appendix E, Figures A.9-A.10"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a promising dataset contribution, but the experimental evaluation requires substantial revision. The most urgent issue is the Table 2 caption/content mismatch: the headline MUG-VOS numbers are presented under a DAVIS-2017 caption, while the appendix gives different DAVIS numbers, so the reader cannot currently verify the main claim. In addition, the test-set-based hyperparameter selection and the lack of significance testing call into question the 0.5-point margin over DEVA. The ground-truth coverage concern is the deepest issue: because the test annotations are constrained to SAM's proposals, the benchmark may not measure what it claims to measure, and MMPM's SAM-based encoder may benefit from this. These issues are fixable within the scope of a major revision if the authors add a proper validation split, report variance or significance, correct the tables, and provide a recall study against exhaustive human annotation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a dataset paper, and the dataset is the contribution. MUG-VOS is 77k clips and 47M masks, an order of magnitude denser than existing VOS benchmarks, and it explicitly targets non-salient, partial, and background masks. That fills a real gap. The collection pipeline—SAM grid prompts, optical flow warping, IoU matching—is simple and clearly described, and the statistics are honestly reported. The test set, built on DAVIS-17 with human accept/reject/refine on SAM-proposed tracks, gives a reasonable starting point for evaluation, though with a significant caveat.\n\nThe biggest issue is that the benchmark can only measure what SAM can propose. Algorithm 1 and the appendix both confirm that human annotators select and refine masks from SAM's output space; they do not discover objects SAM misses. If SAM systematically misses very small, heavily occluded, or odd-granularity objects, they are entirely absent from train and test. Since MMPM uses a pretrained SAM encoder, its 86.1 vs DEVA's 85.6 could reflect distributional familiarity rather than superior tracking. That is a validity threat that needs a human-annotation study on a random sample, measuring coverage against exhaustive segmentation, before the benchmark is taken as ground truth.\n\nOther issues are smaller. Table 2's caption says DAVIS-2017 but the numbers are MUG-VOS; Table A.1 gives different DAVIS numbers. No error bars or significance on 30 videos; the 0.5-point lead over DEVA is within noise. Ablations are done on the test set, which is overfitting the benchmark. No data/code release in the preprint. SAM 2 is not mentioned, which is conspicuous for December 2024.\n\nTo be fair, none of these kill the dataset. The pipeline is reproducible in principle, and the scale is genuinely new. The paper deserves a serious referee whose main job is to push for coverage analysis and data release.\n\nThis is for anyone building open-world or interactive video segmentation models. MMPM is a reasonable baseline but not the story. Engage with it, but require the authors to fix the table caption, report variance, release at least a sample of the data, and measure the pipeline's recall against exhaustive human segmentation. If they cannot show coverage, the dataset should be positioned as pseudo-labeled rather than ground truth.","headline":"A genuinely new large-scale multi-granularity VOS dataset with a clear automatic pipeline, but benchmark validity rests on SAM's proposal recall and the paper has a few sloppy evaluation details.","tokens_in":15282,"tokens_out":2019,"would_cite":false,"duration_ms":18249,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a semi-automatic SAM-plus-optical-flow pipeline can build a large-scale, densely annotated multi-granularity video object segmentation dataset, and that a memory-based model built on SAM's encoder outperforms prior…","keywords":["video object segmentation","multi-granularity masks","semi-automatic annotation","Segment Anything Model","optical flow tracking","memory-based mask propagation","salient and non-salient objects","dense video dataset"],"falsifier":"Independently human-annotate every visible object, part, and stuff region in a random sample of MUG-VOS clips, including objects the SAM pipeline did not propose, and compare the resulting density and track counts with the dataset's; if the pipeline misses or merges non-salient objects at a high rate, the dataset's multi-granularity claim and any model ranking computed on it are compromised.","tokens_in":1948,"feed_emoji":"🎬","tokens_out":2546,"duration_ms":91284,"temperature":0.7,"pith_summary":"Current video segmentation benchmarks annotate only salient foreground objects, so models trained on them fail on non-salient objects, parts, and background stuff. This paper tries to establish that a much broader benchmark is feasible: MUG-VOS, built by running the Segment Anything Model (SAM) on each frame and linking masks across frames with optical flow and IoU matching, contains 77,994 clips and 47 million masks, with a human-verified test set. On this benchmark the paper also proposes MMPM, a model that reuses SAM's image encoder and adds temporal and sequential memory modules to propagate masks through video. MMPM achieves the best reported J&F of 86.1 on the MUG-VOS test set, ahead of 85.6 for DEVA and 83.0 for XMem. If the benchmark is trusted, it provides a way to train and evaluate segmentation of any object, part, or stuff in video, not just the salient ones.","feed_headline":"Memory-augmented SAM tops prior methods on dense video masks","feed_subtitle":"MUG-VOS adds 77,994 clips and 47M masks; the memory-augmented SAM model beats DEVA and XMem.","key_machinery":"The load-bearing machinery is two-part. First, the data collection pipeline: for each starting frame, SAM with grid point prompts produces initial masks; for each next frame, points sampled from the previous target mask are warped by an optical flow map, SAM produces candidate masks from those points, and the candidate whose IoU with the flow-warped previous mask is highest becomes the tracked mask. This turns existing video sources into pseudo-labeled multi-granularity tracks without per-frame human annotation. Second, MMPM: a frozen or fine-tuned SAM image encoder produces query features; a memory-read module computes softmax affinity between query and stored keys; two memories feed the mask decoder, namely temporal memory storing high-resolution key/value pairs from recent frames (capped at Tmax, randomly filtered while keeping first and last frames) and sequential memory storing low-resolution features updated by a GRU. The memory read-out augments the image features so masks can be propagated frame by frame consistently.","core_discovery":"The central claim is that multi-granularity video object segmentation is both measurable and learnable. The paper builds MUG-VOS, at 77,994 clips and roughly 47 million masks the largest and densest video segmentation dataset it compares against (training mask density 0.714, 66.3 masks per frame; test density 0.663, 29.6 masks per frame), with masks spanning salient objects, non-salient objects, parts, and background stuff. The masks come from a pipeline that prompts SAM with grid points on the first frame, samples and warps points with optical flow to later frames, prompts SAM again for candidate masks, and links tracks by maximum IoU between the warped mask and candidates; the test set is human-approved or refined. On this benchmark, the paper's MMPM, a SAM-encoder model with a mask decoder, a high-resolution temporal memory capped at Tmax entries with random filtering that preserves first and last frames, and a low-resolution sequential memory updated by a GRU, reports 86.1 J&F versus 85.6 for DEVA and 83.0 for XMem, with ablations showing both memory types contribute and that update interval r = 5 and N = 10 memory values are sufficient. Retrained on MUG-VOS and evaluated on the DAVIS-17 validation set, MMPM reports 69.1 J&F versus 64.1 for DEVA and 63.0 for XMem.","pith_inferences":["Editorial inference: because the 77,994-clip training set is not human-verified, the benchmark's ground truth inherits SAM's proposal distribution; a model built on SAM, such as MMPM, is therefore evaluated on a distribution shaped by its own backbone.","Editorial inference: a fully manual re-annotation of a sample of MUG-VOS videos, adding any objects SAM did not propose, could shift the reported ranking and would be the direct test of the dataset's coverage claim.","Editorial inference: the same flow-plus-IoU pipeline would likely transfer to other large video corpora, making video-level segment-anything training much cheaper than hand annotation.","Editorial inference: MMPM's two-memory propagation design is not tied to SAM's encoder and could be attached to other promptable image segmentation backbones."],"forward_implications":["MUG-VOS can support training and evaluation of video segmentation models that target non-salient objects, parts, and background stuff, not just salient foreground instances.","Existing VOS methods such as XMem and DEVA, when retrained on MUG-VOS, score well below MMPM, indicating dense multi-granularity masks are a harder and currently underserved regime.","MMPM's memory design, preserving the first and last stored frames, updating every five frames, and keeping ten temporal values, is a workable recipe for extending SAM-like image segmentation to video.","The benchmark's density (0.714 train, 0.663 test) and 47M masks offer supervision comparable in spirit to image-level segment-anything data, potentially helping class-agnostic video understanding.","Because the test set is human-verified, MUG-VOS can serve as a fairer evaluation for methods that must track whatever a user points at, including objects that are not salient."],"supporting_citations":[{"why":"Supplies SAM and SA-1B, the image segmentation model and dense-mask data that the collection pipeline and MMPM encoder build on.","marker":"(Kirillov et al. 2023)"},{"why":"Provides the HD-VILA-100M video corpus from which the 77,994 MUG-VOS clips are drawn.","marker":"(Xue et al. 2022)"},{"why":"Its DAVIS-17 validation videos supply the 30 test videos that receive additional multi-granularity annotations and the transfer benchmark used in Table A.1.","marker":"(Pont-Tuset et al. 2017)"},{"why":"XMem is the memory-based VOS baseline that MMPM is compared against and whose Atkinson-Shiffrin-style memory inspires the temporal memory design.","marker":"(Cheng and Schwing 2022)"},{"why":"DEVA is the SAM-plus-propagation baseline compared against MMPM on MUG-VOS.","marker":"(Cheng et al. 2023)"},{"why":"Provides the affinity/similarity computation (STCN) used in MMPM's memory read-out.","marker":"(Cheng, Tai, and Tang 2021)"},{"why":"UVO is the dense open-world video segmentation dataset that MUG-VOS extends to non-salient and partial objects.","marker":"(Wang et al. 2021)"},{"why":"YouTube-VOS is a salient-only benchmark whose limitations motivate the multi-granularity dataset.","marker":"(Xu et al. 2018)"}],"fun_headline_variants":["New dataset and model push video segmentation beyond salient objects","MUG-VOS: 47M masks help memory model beat DEVA and XMem","Multi-granularity video segmentation: memory model wins on new benchmark","Dense video masks: MUG-VOS dataset and MMPM outperform prior art","Tracking everything: memory-augmented model excels on new MUG-VOS"],"cache_read_input_tokens":17280,"weakest_assumption_plain":"The benchmark's ground truth is only as good as the automatic SAM-and-flow masks, and the 77,994-clip training set is never human-verified, so if this pipeline systematically misses or merges objects, both training and evaluation inherit that blind spot.","fun_headline_variants_meta":{"raw":{"variants":["New dataset and model push video segmentation beyond salient objects","MUG-VOS: 47M masks help memory model beat DEVA and XMem","Multi-granularity video segmentation: memory model wins on new benchmark","Dense video masks: MUG-VOS dataset and MMPM outperform prior art","Tracking everything: memory-augmented model excels on new MUG-VOS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1918,"prompt_tokens":1052,"completion_tokens":866,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":767}},"tokens_in":668,"tokens_out":866,"duration_ms":6687,"temperature":1.0,"reasoning_tokens":767,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:17:54.810818+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently human-annotate every visible object, part, and stuff region in a random sample of MUG-VOS clips, including objects the SAM pipeline did not propose, and compare the resulting density and track counts with the dataset's; if the pipeline misses or merges non-salient objects at a high rate, the dataset's multi-granularity claim and any model ranking computed on it are compromised.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the HD-VILA-100M video corpus from which the 77,994 MUG-VOS clips are drawn."},{"cited_title":"K.; and Schwing, A","cited_arxiv_id":null,"evidence_quote":"XMem is the memory-based VOS baseline that MMPM is compared against and whose Atkinson-Shiffrin-style memory inspires the temporal memory design."},{"cited_title":"K.; Tai, Y.-W.; and Tang, C.-K","cited_arxiv_id":null,"evidence_quote":"Provides the affinity/similarity computation (STCN) used in MMPM's memory read-out."}],"review_version":1}