{"id":"57a02ec5-7900-48da-9d13-1d8246f658b1","arxiv_id":"2507.20120","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Local2Global inserts a lightweight query aligner between encoder and decoder in a DETR-based segmenter, improving temporal consistency and achieving state-of-the-art results on four video segmentation benchmarks.","lead":"This paper introduces Local2Global, an online video instance segmentation method that aligns past tracking queries with current frame features before segmentation, improving temporal consistency across frames. The lightweight alignment module reports new state-of-the-art results on YouTube-VIS, OVIS, and VIPSeg benchmarks with a simple online training setup.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Local-query selection is underspecified: the class-prediction head that scores encoder tokens is never defined or trained, so the L2G-aligner's input is ambiguous and the central claim is not yet verifiable.","rationale":"Good-faith reading: the paper proposes a clean, well-ablated mechanism and reports consistent gains across four benchmarks; the ablation isolating L2G-aligner (+4.6 AP OVIS) is strong internal evidence that the added module matters. The load-bearing risk is not the gains themselves but the explanation of why they occur. The L2G-aligner's input, qlocal, is described as topK encoder tokens scored by classification confidence, yet the paper never defines or trains a token-level classifier. In Mask2Former the class head consumes decoder queries, not encoder spatial tokens; applying it to tokens is not described and no auxiliary loss is introduced. Thus the central 'local-to-global alignment' step is not reproducible from the text. The manuscript's own admission that qlocal contains duplicates/background underscores that this selection mechanism is non-trivial and needs validation. The reader's weakest assumption (faithfulness of selected tokens) is related but slightly weaker; we sharpen it to an undefined forward computation. Other issues (YTVIS-2019 53.4 vs 54.3 typo, no error bars, no code) are real but secondary. Verdict: conditional acceptance, with the condition that the authors specify the token-scoring head, its training supervision, and the value of K, and ideally release code.","tokens_in":14096,"tokens_out":6366,"duration_ms":64076,"concrete_test":"Analytical test: Re-derive the forward pass for qlocal from Section 3.2 using only components defined in Section 3.1. Specifically, compute the output of the class-prediction head P on E(f_t) (shape H·W×c). If P is a linear layer trained on decoder queries, its application to E tokens is well-defined numerically but its confidence values have no training signal. Check whether any loss term in Eq. 4 supervises token-level classification. If no such term exists, the selection confidence is arbitrary. A compelling computational companion: in the OVIS ablation (Table 3), replace the confidence-based topK with K tokens chosen uniformly at random; if AP remains near 37.0, the proposed selection is not load-bearing; if AP drops sharply, the missing head definition is critical to reproduce the claimed mechanism.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The core mechanism, early alignment of global queries with per-frame local queries, depends on qlocal being a faithful set of current-frame object cues. Section 3.2 states that qlocal is obtained from the encoder E 'through the topK∈ N×c token prediction, where the topK is determined based on the classification confidence computed by the class-prediction head for each token.' However, in the Mask2Former backbone described in Section 3.1, the class-prediction head operates on decoder output queries (N vectors), not on the encoder's spatial tokens (H×W vectors, Eq. 1). The paper does not describe any new head, loss, or training procedure that would make per-token confidence meaningful. Without this, the topK selection is undefined: K is not specified, the head's parameters are not specified, and no evidence is given that the selected tokens correspond to instances rather than background. The authors themselves note that qlocal 'may include numerous duplicate or background predictions' (Section 3.2) and handle this only by using static position embeddings. The central claim therefore rests on a component whose forward computation is not derivable from the paper. This is a reproducibility gap, not merely a noise concern: a reader cannot implement L2G-aligner without inventing an auxiliary token-classification module.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Local2Global, an online video instance segmentation framework built on Mask2Former with query propagation. It introduces two query sets: local queries, selected from current-frame encoder features by token-level classification confidence, and global queries, which are the propagated decoder queries from past frames. A lightweight transformer decoder, the L2G-aligner, performs cross-attention with the local queries as key/value and global queries as queries, producing an aligned query representation before the segmentation decoder. The method also propagates trajectory position embeddings computed from the previous frame's predicted masks via Mask2Box, sinusoidal encoding, and an MLP. Training uses standard detection/segmentation losses plus a reduced-supervision scheme that randomly skips intermediate frames when computing losses. Experiments report state-of-the-art results on YouTube-VIS 2019/2021/2022, OVIS, and VIPSeg; the main ablation (Table 3) shows the L2G-aligner adds 4.6 AP on OVIS and 5.0 AP on YTVIS-2021 over the propagated-query baseline, and trajectory embeddings add another 0.8 AP on OVIS.","tokens_in":14368,"tokens_out":5668,"duration_ms":57032,"significance":"If the claims hold, the contribution is significant for online video instance segmentation: a simple, lightweight module integrated directly into the segmentation decoder path improves temporal consistency without external memory banks, separate tracking modules, or multi-stage offline-to-online training. The ablation in Table 3 directly isolates the proposed L2G-aligner and shows a large and consistent gain across two datasets, which is the paper's central evidence. The cross-dataset evaluation (YTVIS-19/21/22, OVIS, VIPSeg) is broad, and the reported parameter overhead (3.2M parameters) is modest. However, the current manuscript has a load-bearing reproducibility gap: the local-query selection mechanism is not fully specified, and the reported YTVIS-2019 number conflicts between the abstract/table and the main text. The absence of code and error bars further tempers confidence, although these are not themselves disqualifying given the size of the reported effects.","major_comments":[{"comment":"The local-query selection mechanism is undefined. The text states that qlocal is obtained from the encoder E \"through the topK∈ N×c token prediction, where the topK is determined based on the classification confidence computed by the class-prediction head for each token.\" In the Mask2Former backbone described in Section 3.1, the class-prediction head operates on the decoder output queries, not on the encoder's spatial tokens. The paper does not define a new token-level classification head, nor its loss, training schedule, or the value of K, nor how the head is applied to the multi-scale encoder features. Because qlocal serves as the key/value input to the L2G-aligner in Eq. (3), the forward computation of the central proposed module cannot be reproduced from the paper as written. Please provide the full specification of this token-classification head, its supervision, K, and the token selection procedure, or replace the mechanism with one that is fully derivable from the described architecture.","section":"Section 3.2 (L2G-aligner paragraph, Eq. (3))"},{"comment":"There is a numerical inconsistency in the headline result. The abstract and Table 1 report 54.3 AP on YouTube-VIS-2019, but Section 4.3 states \"Local2Global achieves 53.4 AP\" and claims it outperforms the closest online method by 3.1 AP. The 3.1 AP margin is consistent only with 54.3 AP against DVIS (51.2 AP), not with 53.4 AP. This discrepancy must be corrected and the correct number used consistently throughout.","section":"Section 4.3 (YTVIS-2019/2021) vs. Abstract and Table 1"},{"comment":"The ablation baseline's use of position embeddings is ambiguous, which matters for interpreting the central claim. Section 3.1 introduces anchor-box dynamic positional embeddings as part of the backbone and states they act as trajectory embeddings, yet Table 3 treats \"trajectory\" as a separate addition after the L2G-aligner. It is therefore unclear whether each row of Table 3 uses static or dynamic position embeddings for the global queries, and whether the baseline already includes the dynamic position embeddings from Eq. (2). Please state explicitly, row by row, which positional embeddings are used, so that the +4.6/+5.0 AP gain attributed to the L2G-aligner is not confounded with the gain from switching to trajectory position embeddings.","section":"Table 3 and Section 4.4 (Effect of the proposed L2G-aligner)"}],"minor_comments":[{"comment":"There are several typos: \"both both positional and visual features\" and \"instance of an instance\" should be corrected, and the phrase \"it comprising\" is ungrammatical.","section":"Section 3.1"},{"comment":"The word \"segementation\" is misspelled; also the expression \"topK∈ N×c token prediction\" is unclear notation and should be rewritten, e.g., as \"the top-K tokens are selected and projected to a set of N query vectors of dimension c.\"","section":"Section 3.2"},{"comment":"The paper states all experiments use a single training run without reporting variance. Please indicate the number of random seeds and, if only one run was performed, state this explicitly in the experimental setup.","section":"Section 4.2"},{"comment":"The phrase \"without bells and rings\" is informal; consider replacing with \"without bells and whistles\" or a standard phrase such as \"with a simple architecture.\"","section":"Abstract and Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising and the ablation evidence is broadly convincing in magnitude, but the underspecified local-query selection is a genuine reproducibility blocker that the authors must resolve before the central mechanism can be verified. The YTVIS-2019 numerical inconsistency also needs correction. Given the emphasis on simplicity and online training, I would encourage the editor to request code release or at least a detailed appendix on implementation, including the token-classification head, K, and training losses for that head."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, well-ablated architectural contribution that deserves a real referee, but the paper's central mechanism is under-specified enough that I couldn't reimplement it from the text.\n\nThe new thing is placing a lightweight cross-attention stage (L2G-aligner) between Mask2Former's encoder and decoder, where it aligns propagated 'global' queries with per-frame 'local' queries, plus explicit trajectory embeddings computed from the previous mask via Mask2Box. The placement matters: unlike TCOVIS's external STE, this is inside the segmenter. The ablation table supports the claim: the aligner adds 4.6-5.0 AP over a propagated-query baseline on OVIS and YTVIS-2021, and trajectory embeddings add another 0.8 AP. Those are large, consistent effects across four benchmarks with a 3.2M parameter cost. The authors also honestly discuss the limitation (reappearing objects after occlusion). The paper is clearly written and the experiments are sensibly designed.\n\nThe soft spot is real. Section 3.2 defines qlocal as the topK encoder tokens ranked by classification confidence from the 'class-prediction head.' In Mask2Former the class head operates on decoder output queries, not on the encoder's spatial tokens. The paper never defines this head, its loss, or the value of K. So the core input to the L2G-aligner is ambiguous. The authors even note qlocal may contain duplicates and background. This is a reproducibility gap, not a fatal conceptual flaw: the architecture could work with an auxiliary token-classification module, but that module is not described. I'd want code or a precise spec before trusting the SOTA numbers.\n\nOther issues are minor. No error bars or multi-seed runs; single-run results on this kind of benchmark should at least report variance. There's also a small typo: the YTVIS-2019 text says 53.4 AP while Table 1 and the abstract say 54.3. And the paper doesn't release code.\n\nOverall: the central idea is plausible and the ablation evidence is strong. The missing detail on local-query selection is the main thing to fix, and it's fixable. This deserves peer review rather than desk rejection.","headline":"A genuinely simple, well-ablated fix for temporal drift in online VIS that deserves peer review, but the local-query selection step is under-specified enough that the main result isn't yet reproducible.","tokens_in":14900,"tokens_out":2448,"would_cite":false,"duration_ms":23711,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Early alignment of propagated global queries with current-frame local queries, via the lightweight L2G-aligner, is claimed to be the key to temporally consistent online video instance segmentation.","keywords":["video instance segmentation","query propagation","temporal consistency","L2G-aligner","DETR object queries","transformer decoder","trajectory position embedding","online video segmentation"],"falsifier":"Measure the recall of the top-K selected local queries against ground-truth instance masks on OVIS frames with heavy occlusion; if instances that later reappear are absent from the selected token set in the frames where they are occluded, then the aligner cannot be the source of their recovery, and the paper's account of how current-frame cues enter the propagated query would fail that test.","tokens_in":13874,"feed_emoji":"🎬","tokens_out":11859,"duration_ms":96408,"temperature":0.7,"pith_summary":"Local2Global's central claim is that in online video instance segmentation, temporally consistent predictions come from aligning the propagated global queries, which carry past object representations, with local queries drawn from the current frame before the segmentation decoder runs, rather than from adding memory banks or separate tracking modules. The paper introduces a lightweight transformer decoder, the L2G-aligner, that performs this alignment, and a trajectory position embedding computed from the previous frame's mask. It reports that adding the aligner raises AP on OVIS from 31.6 to 36.2 and on YouTube-VIS 2021 from 43.9 to 48.9, and that including trajectory propagation reaches 37.0 AP on OVIS and 54.3 AP on YouTube-VIS 2019 with a ResNet-50 backbone. A sympathetic reader would care because the method works with simple online training and no external memory, suggesting that the bottleneck in propagation-based VIS is the quality of the alignment step, not the complexity of the temporal model.","feed_headline":"Early query alignment lifts video instance segmentation to 54.3 AP","feed_subtitle":"Aligning past queries with current-frame cues before decoding stops drift and beats offline methods.","key_machinery":"The L2G-aligner is a lightweight transformer decoder (three layers, hidden dimension 1024, about 3.2 million parameters) placed between the segmentation encoder and decoder. It takes the global queries, which are the final decoder output queries from the previous frame and carry instance identity, as queries, and takes local queries, the current frame's top-K encoder tokens ranked by classification confidence with static learnable position embeddings, as key-value pairs; cross-attention then reorients the past representation toward the current frame. The second mechanism is the trajectory position embedding: the previous frame's predicted mask is converted to a box, encoded with a sinusoidal positional encoding, and projected by an MLP, so the global query's position is updated dynamically along the instance's path. Together the two mechanisms carry the argument: the aligner supplies current visual context, the dynamic position supplies current spatial context, and the decoder then applies its image-domain localization bias.","core_discovery":"On the paper's own terms, the discovery is that query propagation in DETR-style video segmentation degrades because past queries are handed to the segmentation decoder with a spatial bias trained for current-frame images, and the fix is to align those propagating global queries with a compact set of current-frame local queries (the top-K encoder tokens ranked by classification confidence) through a lightweight cross-attention decoder placed between encoder and decoder. Local queries carry the current frame's visual cues without committing to a spatial anchor, global queries maintain instance identity across frames, and the alignment reorients the past representation before the decoder uses it. Explicit trajectory information is then added by computing dynamic position embeddings from the previous frame's mask box, so the query both looks and moves like the instance. The ablations attribute 4.6 to 5.0 AP of the gain to the aligner and a further 0.8 AP on OVIS to trajectory embeddings, leading to 54.3 and 49.4 AP on YouTube-VIS 2019 and 2021, 42.3 AP on YouTube-VIS 2022, 37.0 AP on OVIS, and 46.3 VPQ on VIPSeg.","pith_inferences":["A testable extension is replacing the confidence-based top-K selection with a learned or mask-guided proposal step; the paper's own limitation section reports failures when objects disappear and reappear, which is exactly the regime where confidence-selected tokens are most likely to miss the object.","If the alignment hypothesis generalizes, the same early-alignment step should transfer to other DETR-family propagation pipelines, such as video object detection or video object segmentation, with a similar reduction in drift.","The static-position choice for local queries suggests a broader design principle for query-based video models: do not attach dynamic, instance-specific position embeddings to unrefined candidate tokens; apply them only after alignment has anchored the token.","Stress-testing the claim on a dataset with heavy occlusion and object reappearance, such as the MOSE dataset the paper itself names, would locate the boundary of the alignment mechanism."],"forward_implications":["Propagating raw queries through the decoder without alignment is the identified failure mode; the paper's baseline loses 4.6 to 5.0 AP compared with the same model using the L2G-aligner.","Online video instance segmentation does not need separate memory banks, decoupled temporal transformers, or offline-then-online training to match or beat offline methods on the tested benchmarks.","The segmentation decoder's pretrained frame-level bias is a reusable asset for video, provided past queries are re-anchored to current-frame cues before decoding.","Explicitly propagating where an instance was, through dynamic position embeddings from the previous mask, is worth about 0.8 AP on the heavily occluded OVIS dataset.","Static rather than dynamic position embeddings for local queries is the right design choice, because unrefined local tokens contain duplicate and background predictions."],"supporting_citations":[{"why":"It supplies the Mask2Former encoder-decoder backbone that Local2Global extends, with the aligner inserted between its encoder and decoder.","marker":"[10]"},{"why":"It establishes DETR's object-query mechanism and the spatial bias that the paper identifies as the source of drift in propagation.","marker":"[6]"},{"why":"It is the GenVIS query-propagation framework with memory that Local2Global compares against and improves across the reported benchmarks.","marker":"[18]"},{"why":"It provides InstanceFormer, an online query-propagation VIS framework whose propagation scheme Local2Global inherits.","marker":"[24]"},{"why":"It supplies the Mask2Box operation and the dynamic positional embedding used to compute trajectory position from the previous frame's mask.","marker":"[25]"},{"why":"It provides the DAB-DETR anchor-box dynamic positional embedding that the trajectory embedding is built on.","marker":"[32]"}],"fun_headline_variants":["Early query alignment lifts video segmentation to 54.3 AP","Align local and global queries to stop video segmentation drift","L2G-aligner: a tiny decoder aligns queries for stable video segmentation","Pre-decoder query alignment beats drift in DETR video segmentation","54.3 AP: early query alignment keeps video segmentation on track"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the few current-frame feature entries with the highest classification scores are a faithful and sufficient summary of every object in the frame, including occluded and never-before-seen objects; if that selection misses an instance, the aligner can only inject noise into that instance's global query.","fun_headline_variants_meta":{"raw":{"variants":["Early query alignment lifts video segmentation to 54.3 AP","Align local and global queries to stop video segmentation drift","L2G-aligner: a tiny decoder aligns queries for stable video segmentation","Pre-decoder query alignment beats drift in DETR video segmentation","54.3 AP: early query alignment keeps video segmentation on track"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000917,"raw_usage":{"total_tokens":3977,"prompt_tokens":1027,"completion_tokens":2950,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":2861}},"tokens_in":643,"tokens_out":2950,"duration_ms":19360,"temperature":1.0,"reasoning_tokens":2861,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:48:46.592704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the recall of the top-K selected local queries against ground-truth instance masks on OVIS frames with heavy occlusion; if instances that later reappear are absent from the selected token set in the frames where they are occluded, then the aligner cannot be the source of their recovery, and the paper's account of how current-frame cues enter the propagated query would fail that test.","supporting_citations":[{"cited_title":"In: NeurIPS (2021) 1, 2, 3","cited_arxiv_id":null,"evidence_quote":"It supplies the Mask2Former encoder-decoder backbone that Local2Global extends, with the aligner inserted between its encoder and decoder."},{"cited_title":"In: ECCV (2020) 1, 2, 8","cited_arxiv_id":null,"evidence_quote":"It establishes DETR's object-query mechanism and the spatial bias that the paper identifies as the source of drift in propagation."},{"cited_title":"In: CVPR (2023) 1, 2, 3, 4, 5, 6, 7, 11","cited_arxiv_id":null,"evidence_quote":"It is the GenVIS query-propagation framework with memory that Local2Global compares against and improves across the reported benchmarks."},{"cited_title":"In: Pro- ceedings of the AAAI Conference on Artificial Intel- ligence","cited_arxiv_id":null,"evidence_quote":"It provides InstanceFormer, an online query-propagation VIS framework whose propagation scheme Local2Global inherits."},{"cited_title":"In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"It supplies the Mask2Box operation and the dynamic positional embedding used to compute trajectory position from the previous frame's mask."}],"review_version":1}