{"id":"1ae5b3e4-2b6c-4948-87fd-02c63cba60ea","arxiv_id":"2507.18921","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HQ-SMem wraps DeAOT-family trackers with SAM-HQ mask refinement, DINO-validated acceptance, and selective keyframe memory to improve mask quality and long-video efficiency.","lead":"This paper adds a memory-management and mask-refinement layer on top of existing video object segmentation models, using SAM-HQ and DINO features to improve masks and store only key frames. It reports consistent quality gains on long-video and object-transformation benchmarks, including new top scores on Long Video Dataset and LVOS.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Memory-efficiency claim conflates selective eviction with a fixed L/30 update rate; the only ablation isolating S-Mem shows a ~0.3% Q gain, leaving the title's key memory contribution unvalidated.","rationale":"The reader's weakest assumption concerns DINO-based verification in Eq. (9): if DINO cosine similarity cannot distinguish the tracked object from a similar neighbor, wrong SAM masks could be accepted and propagated. That is a plausible failure mode, but the paper itself acknowledges related limitations (Supplementary Section F: instance confusion under occlusion and similar objects), and the SAM-HQTF ablation shows the mechanism does help on aggregate. The more load-bearing concern is the memory-efficiency claim, which is central to the paper's title and to Figure 4. The supplementary protocol explicitly fixes the update frequency to L/30, so the near-constant memory curve is at least partly a consequence of subsampling rather than of the selective removal rule. The only ablation isolating S-Mem shows a Q gain of 0.003, so the selective mechanism contributes little to accuracy, while its contribution to memory is never tested at a matched update rate. This does not disprove the empirical results, but it means the paper's headline memory contribution is unvalidated as stated. The appropriate verdict remains CONDITIONAL: the authors should release code, state tau and lambda, add error bars, and provide a matched-frequency memory ablation. My read does not move the verdict, hence UNCHANGED.","tokens_in":16255,"tokens_out":4479,"duration_ms":50235,"concrete_test":"Run DeAOT on VOTS2024 and LVOS with the same L/30 memory-update cadence used in C.3, but with a non-selective eviction rule (e.g., FIFO or random drop) instead of Eqs. (5)-(7), keeping all other settings identical. If memory usage is already near-constant and Q is within 0.3% of the HQ-SMem result, then the selective removal criterion is not responsible for the memory-efficiency or quality gains, and the memory claim in the title and Figure 4 must be revised to a subsampling-rate effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central memory claim—near-constant memory with compact selective key-frame storage (Figure 4)—is confounded by the evaluation protocol in Supplementary Section C.3, which sets the memory bank update frequency to L/30 for VOTS, LVOS, and Long Video Dataset. With updates only every L/30 frames, even a non-selective FIFO or uniform subsampling policy would cap memory size at roughly 30 entries regardless of Eqs. (5)-(7). The baseline DeAOT is compared with a memory bank updated every frame, so the flat memory curve in Figure 4 may reflect update frequency rather than the selective removal scoring function O(z_j) in Eq. (6). The component ablation in Supplementary Table 1 shows that adding S-Mem alone improves Q from 0.636 to 0.639 and Acc from 0.751 to 0.758, while adding SAM-HQTF contributes the larger gains (Q 0.639 to 0.654, Acc 0.758 to 0.788). This indicates that selective memory, as isolated, is responsible for only a small fraction of the reported quality improvement, while the large memory reduction is not isolated at matched update frequency. The claimed 'smart memory' contribution is therefore not established as the cause of the memory scaling shown in Figure 4.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HQ-SMem, an inference-time wrapper around DeAOT/DMAOT VOS trackers. Coarse tracker masks are used to prompt HQ-SAM, the resulting SAM proposals are accepted or rejected via DINO feature similarity (Eqs. 8-9), accepted refined masks are fed back into the tracker as an online teacher-forcing prior (Eq. 10), and the tracker memory is managed by a relevance-freshness score with conditional eviction (Eqs. 3-7). Experiments on VOTS2024, VOTSt, Long Video Dataset, and LVOS report consistent Q/Acc gains over the base trackers, a top-two VOTS placement, and an approximately flat memory curve. The paper also includes a component ablation, a prompt ablation, and a limitations section.","tokens_in":16594,"tokens_out":6570,"duration_ms":72438,"significance":"The evaluation uses external held-out benchmarks and pretrained external components, and the multi-backbone ablation in Table 4 shows consistent gains of roughly 2.4-7.6% in Q across DeAOT/DMAOT variants, which is a real strength. The component ablation in the supplement indicates that the HQ-SAM teacher-forcing path is the main quality driver, and the manuscript is transparent about failure cases in the limitations section. If the memory-efficiency and benchmark claims are substantiated, HQ-SMem would be a useful plug-and-play enhancement for existing trackers. However, the memory-efficiency claim is currently confounded by the fixed L/30 update schedule in the evaluation protocol, and the 'new benchmark' claim is made against a dated comparison set.","major_comments":[{"comment":"The memory-efficiency claim is confounded by the evaluation protocol. Supplementary C.3 states that for VOTS, LVOS, and Long Video Dataset the memory bank update frequency is set to L/30; with updates only once every L/30 frames, even a non-selective uniform-subsampling policy would cap the bank at roughly 30 entries, so the flat blue curve in Fig. 4 does not demonstrate the benefit of the selective scoring function O(z_j) in Eq. (6). The component ablation in Supplementary Table 1 isolates S-Mem at Q 0.636 to 0.639 and Acc 0.751 to 0.758, a gain of only +0.3 Q points, whereas the teacher-forcing component accounts for the larger gains (Q 0.639 to 0.654, Acc 0.758 to 0.788). Please report memory size and quality under a matched update frequency for the base tracker and for a non-selective subsampling baseline, so that the selective-update contribution can be evaluated at fixed update frequency.","section":"Supplementary C.3, Fig. 4, Supplementary Table 1"},{"comment":"The update rule in Eq. (7) does not guarantee the bounded memory that Section 3.2 and Figure 4 claim. When Rel(z_del, z_t) < tau, the rule appends z_t without deleting any frame, so the memory bank can grow without bound if consecutive frames are sufficiently dissimilar. This means the theoretical 'near-constant scaling' statement is not supported by the stated algorithm. Please either prove a bound under the threshold conditions, modify the rule so that a frame is always evicted, or report the actual bank sizes on the long-video benchmarks.","section":"Eq. (7)"},{"comment":"The abstract's claim that HQ-SMem 'sets new benchmarks' on Long Video Dataset and LVOS is not supported by the comparison set in Table 2. The table lists methods up to XMem and RMem and does not include recent published VOS methods evaluated on these benchmarks, such as Cutie, SAM 2, and other 2023-2025 long-video trackers. Without comparisons against the current leaderboard, the 'new benchmark' claim is unsubstantiated; please either add these comparisons or temper the claim to 'improves over the compared baselines.'","section":"Tables 2(a)-(b), Abstract"},{"comment":"The method depends on several hand-set hyperparameters, notably the DINO acceptance threshold tau in Eq. (9) and the relevance-freshness trade-off lambda in Eq. (5), but no sensitivity analysis or selection criterion is reported for them. Since these thresholds directly control which SAM proposals are accepted and which frames are evicted, a sensitivity study or at least the chosen values with a justification is needed to establish that the reported gains are robust rather than tuned to the selected benchmarks.","section":"Eqs. (5) and (9), Supplementary Table 2"}],"minor_comments":[{"comment":"The heading contains a typo, 'dtate-of-the-art', which should read 'state-of-the-art'.","section":"Section 4.5"},{"comment":"The notation p^c_{i,t} in Eq. (9) is inconsistent with p_{t,i} in Eq. (8), and the superscript c is never defined.","section":"Eqs. (8)-(9)"},{"comment":"The text says experiments are run on DAVIS and YouTubeVOS, but in the experimental setup these datasets are used as training sets rather than evaluation benchmarks; the wording should be revised for accuracy.","section":"Section 2.4"},{"comment":"The learning-rate decay schedule is given as a polynomial with power p, but the value of p is not specified.","section":"Supplementary C.2"},{"comment":"The phrase 'self-supervised distillation feedback' is not an accurate description of the online teacher-forcing mechanism, which feeds HQ-SAM refined masks back into the tracker rather than performing distillation; consider renaming this component.","section":"Title and Section 3.3"},{"comment":"The qualitative comparison would be more informative with larger crops and explicit annotations marking the boundary differences described in the text.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core quality-improvement results appear credible; the main risk is overclaiming the memory-efficiency contribution, which is fixable with matched-update-frequency experiments and a corrected update rule. I do not see grounds for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent empirical wrapper paper, not a breakthrough. The most valuable result is the ablation: naively running DeAOT masks through SAM-HQ drops Q from 0.636 to 0.622, while the DINO-validated version with teacher forcing raises it to 0.654. That one comparison justifies the design — SAM prompt ambiguity is real, and feature-based validation fixes it. The full system improves Q by roughly 2-8% across twelve DeAOT/DMAOT variants, and the Long Video / LVOS numbers lead the methods they compare against. The package is new even if no single component is: SAM-HQ refinement, DINO verification, selective memory, and online mask feedback in one loop, evaluated on external held-out benchmarks.\n\nNow the soft spots, in proportion.\n\nThe title's memory claim is not established. Supplementary C.3 sets the memory-bank update frequency to L/30 for VOTS, LVOS, and Long Video. That cadence alone caps memory at tens of entries regardless of what the scoring function in Eq. (6) does, and the DeAOT baseline is fed a new frame every frame. Figure 4's flat curve is explained by update frequency, not smart eviction. The ablation supports this reading: S-Mem alone adds 0.003 Q; the teacher-forcing component adds 0.015. The paper needs a FIFO baseline at the same update cadence before it can claim memory-efficient selective storage.\n\nSecond, reproducibility: λ, τ_mem, and τ_dino never get values, code is only promised on acceptance, and there are no error bars. The Long Video margin over RMem is 0.8 J&F points — thin without variance. \"New benchmarks\" is also too strong given the partial comparison sets; and on LVOS, AOT+HQ-SMem (65.1) loses to AOT+RMem (66.1), which quietly undercuts the memory narrative.\n\nThe DINO-acceptance assumption the reader worries about — that high cosine similarity between coarse mask and SAM proposal means the same object — is a genuine risk, but the paper's own ablation (naive SAM hurts, validated SAM helps) is evidence in its favor. I'd call it a moderate concern, not a load-bearing flaw. Credit where due: the supplementary limitations section openly lists drift, instance confusion, and the ten-object cap.\n\nWho this is for: VOS practitioners who want a tested inference-time wrapper around DeAOT-class trackers. It deserves a serious referee; I'd send it to review with a demand for matched-cadence memory baselines, hyperparameter values, and code or error bars. The core empirical signal looks real; the memory scaffolding is the weak floorboard.","headline":"A genuine plug-in gain for DeAOT-class trackers, with the strongest evidence in the validation ablation — but the memory-efficiency headline rests on an update-rate confound, not on the selective-eviction rule.","tokens_in":17128,"tokens_out":4772,"would_cite":true,"duration_ms":48579,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HQ-SMem wraps existing video object segmentation trackers in a SAM-HQ refinement loop gated by DINO-feature similarity, with selective memory pruning and online teacher forcing, to sharpen mask boundaries, resist drift, and keep memory…","keywords":["video object segmentation","SAM-HQ","DINO feature similarity","smart memory","teacher forcing","long-term video","topological transformation","DeAOT"],"falsifier":"Run HQ-SMem on a sequence with two similar objects that cross or on a topology-split sequence, sweeping the acceptance threshold from 0 to 1, and compare quality and accuracy against the base DeAOT tracker; the paper's claim fails if no threshold gives a consistent win on both split-object and long-video sets. A more direct check is to measure the DINO similarity between true target masks and impostor masks across VOTSt and LVOS: if the two distributions overlap substantially, no threshold can implement the same-object filter the argument assumes.","tokens_in":16101,"feed_emoji":"🎬","tokens_out":9486,"duration_ms":92048,"temperature":0.7,"pith_summary":"This paper claims that an existing video object segmentation tracker can be upgraded without retraining it: coarse masks from DeAOT or DMAOT are sent to the high-quality SAM-HQ model, and a DINO-feature similarity check decides whether the refined mask is the same object. The accepted masks are fed back into the tracker's decoder as the next frame's prior, which the authors call online teacher forcing, and a smart memory keeps only non-redundant key frames. The claimed payoff is sharper boundaries, less drift under topological changes and occlusions, and a memory footprint that stays nearly flat instead of growing linearly with video length. On VOTS2024 and VOTSt the method ranks among the top two, and on the Long Video Dataset and LVOS it reports the best region similarity and contour accuracy, about 3 percent above the base DeAOT tracker.","feed_headline":"SAM-gated refinement lifts video segmentation by 3 percent","feed_subtitle":"HQ-SMem wraps existing trackers: sharper boundaries, near-constant memory across long and topologically changing videos.","key_machinery":"The load-bearing mechanism is the DINO similarity gate in Eqs. (8)-(9): SAM-HQ yields three proposals per object; the tracker picks the proposal whose DINO feature embedding is closest in cosine similarity to the coarse VOS mask, but accepts it only if the similarity exceeds the threshold $\\tau$; otherwise the coarse mask is kept. This gate is what turns SAM's high-resolution edges into a safe improvement rather than a source of object confusion. The companion smart-memory scorer in Eqs. (3)-(7) deletes the stored frame that maximizes relevance times a freshness penalty, and the teacher-forcing update in Eq. (10) makes every accepted refinement the prior for the next frame.","core_discovery":"The central discovery is that mask quality and memory efficiency can be improved at inference time by gating a strong segmenter with a self-supervised appearance check. HQ-SMem uses the coarse object masks of DeAOT/DMAOT as prompts to SAM-HQ, producing three mask candidates per object; it then chooses the candidate most similar in DINO feature space to the original VOS mask and replaces the mask only when the cosine similarity is above a threshold. This prevents SAM's prompt ambiguity from pulling the mask onto a neighboring object. The selected mask becomes the decoder's prior for the next frame, so the refinement compounds. In parallel, a selective memory update scores each stored frame by relevance to the current frame and age, removing the frame that is both most similar and oldest, which keeps memory nearly constant. The reported consequence is a consistent gain in quality and accuracy across several DeAOT/DMAOT variants, with a 3 percent improvement in J and F on topology-changing data and new best results on long-video benchmarks.","pith_inferences":["We infer that the smart-memory rule could likely be grafted onto other memory-based VOS trackers such as XMem or RMem, since it only uses frame embeddings those models already store; the paper does not test that transfer.","The acceptance threshold is a single global constant, so a per-object or per-sequence threshold calibrated from the first frame's similarity distribution is a plausible robustness improvement, especially for videos where objects become more alike over time.","The paper reports quality and memory but not wall-clock time; the added SAM-HQ and DINO forward passes per frame may dominate runtime, so real-time deployment is an open question.","The same DINO-gated refinement idea could be used in prompt-based video editing, where rejecting mask proposals that leave the object's appearance manifold could reduce mask-stitching errors."],"forward_implications":["On topology-change videos, the base tracker's J and F scores rise by about 3 percentage points when wrapped by HQ-SMem.","On the Long Video Dataset and LVOS, HQ-SMem reports the best region similarity and contour accuracy among the compared methods, including existing SAM-augmented trackers.","The smart memory keeps the end-of-video memory size nearly constant while the base DeAOT's memory grows linearly with frame count, so longer sequences need less extra storage.","The full configuration beats both naively passing masks to SAM (which lowers quality) and using only the selective memory, so the DINO gate is necessary for the reported gain.","The improvement transfers across the DeAOT and DMAOT family: average quality improves by 4.9 percent and accuracy by 5.6 percent over twelve base configurations."],"supporting_citations":[{"why":"DeAOT/DMAOT are the base trackers whose coarse masks HQ-SMem refines and whose scores the ablation table reports.","marker":"[43]"},{"why":"HQ-SAM generates the high-quality mask proposals that the DINO gate selects among.","marker":"[13]"},{"why":"DINO supplies the self-supervised appearance features used to validate SAM proposals and to score frame redundancy.","marker":"[3]"},{"why":"SAM is the promptable segmenter behind the proposal step and the source of the prompt-ambiguity problem the gate addresses.","marker":"[14]"},{"why":"Defines the VOTS2024 and VOTSt benchmarks and the quality/accuracy metrics that anchor the main comparisons.","marker":"[16]"},{"why":"XMem is the long-term memory approach whose linear memory growth motivates the selective update, and a comparison baseline.","marker":"[4]"},{"why":"RMem is the restricted-memory baseline that the paper compares against and extends in training setup.","marker":"[50]"},{"why":"LVOS provides the long-term multi-object benchmark where HQ-SMem reports new state-of-the-art J&F.","marker":"[11]"},{"why":"Long Video Dataset provides the long-sequence benchmark for the reported J and F gains.","marker":"[18]"}],"fun_headline_variants":["Selective memory update lifts video segmentation quality by 3%","Smart memory keeps video tracking sharp on long sequences","HQ-SMem refines video masks with SAM and selective update","Self-supervised feedback improves video object segmentation","Memory-efficient video segmentation with selective frame storage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a high DINO cosine similarity between a coarse tracker mask and a SAM-HQ proposal always means the proposal is a better mask for the exact same object; if the similarity score cannot separate the tracked object from a look-alike neighbor, or lags behind a sudden split or scale change, the gate admits a wrong mask and the teacher-forcing loop propagates the error.","fun_headline_variants_meta":{"raw":{"variants":["Selective memory update lifts video segmentation quality by 3%","Smart memory keeps video tracking sharp on long sequences","HQ-SMem refines video masks with SAM and selective update","Self-supervised feedback improves video object segmentation","Memory-efficient video segmentation with selective frame storage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1656,"prompt_tokens":1032,"completion_tokens":624,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":549}},"tokens_in":648,"tokens_out":624,"duration_ms":6828,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:05:34.678390+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HQ-SMem on a sequence with two similar objects that cross or on a topology-split sequence, sweeping the acceptance threshold from 0 to 1, and compare quality and accuracy against the base DeAOT tracker; the paper's claim fails if no threshold gives a consistent win on both split-object and long-video sets. A more direct check is to measure the DINO similarity between true target masks and impostor masks across VOTSt and LVOS: if the two distributions overlap substantially, no threshold can implement the same-object filter the argument assumes.","supporting_citations":[{"cited_title":"Decoupling features in hierar- chical propagation for video object segmentation","cited_arxiv_id":null,"evidence_quote":"DeAOT/DMAOT are the base trackers whose coarse masks HQ-SMem refines and whose scores the ablation table reports."},{"cited_title":"Segment anything in high quality","cited_arxiv_id":null,"evidence_quote":"HQ-SAM generates the high-quality mask proposals that the DINO gate selects among."},{"cited_title":"Emerg- ing properties in self-supervised vision transformers","cited_arxiv_id":null,"evidence_quote":"DINO supplies the self-supervised appearance features used to validate SAM proposals and to score frame redundancy."},{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"SAM is the promptable segmenter behind the proposal step and the source of the prompt-ambiguity problem the gate addresses."},{"cited_title":"The second visual object tracking seg- mentation vots2024 challenge results, 2024","cited_arxiv_id":null,"evidence_quote":"Defines the VOTS2024 and VOTSt benchmarks and the quality/accuracy metrics that anchor the main comparisons."},{"cited_title":"XMem: long-term video object segmentation with an Atkinson-Shiffrin mem- ory model","cited_arxiv_id":null,"evidence_quote":"XMem is the long-term memory approach whose linear memory growth motivates the selective update, and a comparison baseline."},{"cited_title":"RMem: Re- stricted Memory Banks Improve Video Object Segmentation","cited_arxiv_id":null,"evidence_quote":"RMem is the restricted-memory baseline that the paper compares against and extends in training setup."},{"cited_title":"Lvos: A benchmark for long-term video object segmentation","cited_arxiv_id":null,"evidence_quote":"LVOS provides the long-term multi-object benchmark where HQ-SMem reports new state-of-the-art J&F."},{"cited_title":"Video object segmentation with adaptive feature bank and uncertain-region refinement","cited_arxiv_id":null,"evidence_quote":"Long Video Dataset provides the long-sequence benchmark for the reported J and F gains."}],"review_version":1}