Pith. sign in

REVIEW 4 major objections 5 minor 58 references

The paper argues that prompted and unprompted video segmentation are one problem — sequential mask prediction — and builds a single autoregressive model that beats prior universal streaming methods on seven benchmarks with up to 2.5× faster

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A single autoregressive architecture with a Mamba-based history compressor unifies prompted and unprompted video segmentation and beats prior universal streaming models on seven benchmarks.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid unification paper with real benchmark gains; the 'arbitrary length' selling point outruns the evidence. the 4 major comments →

arxiv 2508.19242 v1 pith:RVGJYAVY submitted 2025-08-26 cs.CV

Autoregressive Universal Video Segmentation Model

Miran Heo , Sukjun Hwang , Min-Hung Chen , Yu-Chiang Frank Wang , Albert Gu , Seon Joo Kim , Ryo Hachiuma This is my paper
classification cs.CV
keywords autoregressive video segmentationuniversal segmentation modelvideo object segmentationvideo instance segmentationstate-space modelsMambaparallel trainingconstant-memory inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that streaming video segmentation is, at bottom, a language-modeling problem: predict each frame's masks conditioned on the current frame, all earlier frames, and all earlier masks, the way an LLM predicts the next word. To make that work it builds AUSM, one architecture in which a Mamba-based History Compressor folds the entire past into a single fixed-size spatial state, so memory stays constant no matter how long the video; a History Marker preserves mask detail by dissolving previous masks into the feature stream instead of compressing each object to a vector. The same design makes training parallel across frames — up to 2.5× faster than iterative training on 16-frame clips — and one set of weights handles both prompted segmentation (track a user-specified object) and unprompted segmentation (discover and track all objects). On seven standard benchmarks AUSM outperforms prior universal streaming video segmentation models. If the framing holds, the fragmented landscape of task-specific video models could consolidate behind a single scalable, LLM-style recipe.

Core claim

Recasting streaming video segmentation as sequential mask prediction — next-frame masks, conditioned on the current frame and on all previous masks, in the same way an LLM predicts the next token — AUSM is a single architecture whose weights serve both prompted video object segmentation (track a user-specified target from an initial mask) and unprompted video instance segmentation (detect and track every object with no external cue). The design replaces the explicit memory buffers of prior VOS systems with a History Compressor: a Mamba-based state-space stack that folds all past frames and masks into one fixed-size spatial state, while a History Marker dissolves previous instance masks direc

What carries the argument

The History Compressor is the load-bearing object: a stack of layers that each mix temporally with a Mamba state-space layer (pixel-wise through time) and spatially with self-attention (frame-independent across pixels), followed by an FFN. Its two properties carry the argument: in recurrent inference, the entire past collapses into one fixed-size spatial state, so memory does not grow with video length; in the parallel training form, the same state is computed teacher-forced over a whole clip, so time cost grows slowly with sequence length. It is paired with the History Marker, which uses Token Mark to add previous masks into the frame-feature stream and thereby avoids the fine-grained infor

Load-bearing premise

That a single fixed-size Mamba state, updated once per frame, retains enough fine-grained mask and identity detail from the entire past that accuracy on very long videos holds up — the paper itself reports degradation on extremely long sequences.

What would settle it

Feed a long video in which a tracked object exits the frame for 100, 500, and 1000 frames and then returns; compare mask IoU and identity-switch rate against a memory-buffer baseline. If accuracy collapses as the gap grows — or if a one-frame appearance buried deep in an uninformative stream is forgotten — the fixed-size-state premise fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • One set of shared weights covers both prompted and unprompted segmentation, selected at inference time by how the ID-vector buffer and mask memory are initialized — no task-specific fine-tuning.
  • Inference memory stays constant as video length grows, so arbitrarily long streams can be processed without the FIFO frame buffers used by STM-style and SAM2-style models.
  • Training is parallel across frames and the speedup grows with sequence length (1.6× at 4 frames to 2.5× at 16), removing a bottleneck for learning from long video.
  • Longer training clips deliver the biggest gains on the hardest multi-object, occlusion-heavy benchmarks (MOSE +4.5, OVIS +5.2), where past context matters most.
  • Repeating or replaying a video at inference refines predictions and improves accuracy — the segmentation analogue of LLM test-time compute scaling (e.g., +1.0 AP on YouTube-VIS 2019).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the LLM analogy is taken seriously, the field's existing toolkit — context-length extrapolation, retrieval-augmented recall (needle-in-a-haystack probes), and inference-time compute scaling — transfers wholesale to video; the paper demonstrates only the last of these and gestures at the first two.
  • The remaining gap to specialized memory-heavy VOS models points to a specific cost of unification: coarse stride-8 frame features trade mask detail for memory savings. A video-specialized backbone with finer features is the natural next experiment.
  • The quadrant-scan repetition trick suggests a general principle beyond repetition — a geometric tour of the visual input acts as a self-correcting refinement loop — which could be extended to other scanning paths or attention-guided tours.
  • A hybrid memory design that pairs the fixed-size Mamba state with a small bounded retrieval cache would directly test whether the compression ceiling, not the architecture, is what limits extremely long sequences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. AUSM recasts streaming video segmentation as autoregressive mask prediction, formalized in Eq. (2). The architecture combines a History Marker that dissolves past instance masks into spatial frame features, a Mamba-based History Compressor that maintains a fixed-size spatial state, a History Decoder, and a Mask2Former-style Pixel Decoder. A single set of weights handles both prompted video object segmentation (VOS) and unprompted video instance segmentation (VIS), with task-dependent initialization of ID vectors and mask buffers. Training is parallelized across frames via teacher forcing (Alg. 2). The paper reports results on DAVIS17, MOSE, YouTube-VOS 2018/2019, YouTube-VIS 2019/2021, and OVIS, claiming that AUSM outperforms prior universal streaming video segmentation models and achieves up to 2.5x faster training on 16-frame sequences. The authors also state in Sec. 5 that performance degrades on extremely long sequences.

Significance. If the results hold, AUSM is a valuable contribution: it unifies prompted and unprompted video segmentation in one architecture, avoids explicit FIFO memory, and introduces a parallel-training scheme with a plausible scaling advantage over recurrent training. The evaluation spans seven benchmarks and both Swin-T and Swin-B backbones. The central risks are that the 'arbitrary length' claim is not supported by a long-horizon experiment, and that the recurrent state is trained only under teacher forcing while inference consumes the model's own mask predictions. The paper's strengths are its clean task unification, modular design, and broad benchmark coverage; its weaknesses are the absence of code or error bars and the under-specified speedup baseline.

major comments (4)
  1. [Sec. 2.2 / Sec. 5] The abstract and Sec. 2.2 claim that AUSM 'scales to video streams of arbitrary length' via a fixed-size Mamba state, but the evidence does not establish this. Training uses 16-frame clips at most (Sec. 3.2 Stage 3), and Sec. 5 concedes 'performance degradation on extremely long sequences.' No experiment measures accuracy as a function of sequence length; OVIS is the longest benchmark but is not a long-horizon stress test. Please add a dedicated long-sequence evaluation (e.g., error vs. video length, or a long-video benchmark) or qualify the claim to 'constant-memory processing' rather than unbounded-horizon performance.
  2. [Sec. 2.3 / Alg. 1-2] There is a train/inference mismatch in the recurrent state. In training (Alg. 2, Preprocess), the History Marker input M_t is built from ground-truth masks, while at inference (Alg. 1, line 19) M_t is updated using the model's own mask predictions. The Mamba state is therefore never trained under the distribution it sees at test time after the first frame. This exposure bias is particularly relevant to the long-horizon claim. Please quantify its effect (e.g., a variant trained with predicted-mask feedback, or scheduled sampling) or provide a clear argument that the discrepancy is benign.
  3. [Sec. 3.4 / Fig. 4] The 2.5x training-speedup claim is central but the 'iterative training' baseline is under-specified. To make the comparison meaningful, state whether the recurrent baseline uses the same architecture with frame-by-frame updates, the same batch size and GPU, and give wall-clock time over repeated runs rather than a single sec/iter value. Without this, the speedup may reflect implementation choices rather than the parallel formulation.
  4. [Table 1] All benchmark numbers are single-run. Several margins over prior universal streaming models are small (e.g., 0.2-1.3 AP on YTVIS21/OVIS in the Swin-B row). Reporting means and standard deviations over multiple seeds, or at least stating the evaluation protocol, would make the 'outperforms' claim statistically supportable.
minor comments (5)
  1. [Intro] The text has missing spaces/OCR artifacts ('trainedonmassivecorporacansubsumediversetasks') in the Introduction; please proofread.
  2. [Sec. 3.4] Typo: 'History Comperssor' should be 'History Compressor'.
  3. [References] The GPT-3 reference [29] is cited as 'Mann et al. (2020)'; the standard attribution is Brown et al. (2020). Please correct.
  4. [Appendix A] The spatial traversal strategy is said to yield 'statistically significant' improvements, but no statistical test is reported. Either report the test or soften the claim.
  5. [Table 1] Swin-L results for AUSM are not reported, although smaller backbones are. A sentence explaining this omission would be helpful.

Circularity Check

0 steps flagged

No significant circularity: the autoregressive framing is a generic factorization, the architecture is specified in explicit equations, and the main results are measured against external benchmarks.

full rationale

I walked the claimed derivation chain: Eq. (2) is a general factorization of video segmentation as sequential mask prediction; it does not by itself determine any segmentation output, and no later equation reduces to it by construction. The History Marker operation is defined explicitly in Sec. 2.2 (the sum over allocated vectors and masks), and the History Compressor is a concrete Mamba/attention/FFN module; these are architectural choices, not fitted parameters renamed as predictions. The training-speedup claim is a measured quantity (Fig. 4), not a consequence of the autoregressive formulation alone. The self-citations to Token Mark (Heo et al., 2025) and RoCoVIS (Heo et al., 2025) are used as component descriptions and background; the load-bearing History Marker behavior is stated in this paper's own equations, so the citations are not the evidence for AUSM's performance. Benchmark results (Tables 1, 3) are against external datasets and prior methods, and the foreground threshold in Table 2 is an inference hyperparameter chosen for robustness, not a fitted target. The paper's own Sec. 5 limitation—degradation on extremely long sequences—is an evidence gap for the 'arbitrary length' wording, but that is a correctness/extrapolation concern, not circularity. No equation or fitted parameter in the paper reduces to its own input.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The paper's central empirical contributions rest on standard deep learning engineering choices (backbone, optimizers, dataset mixing) plus two architectural assumptions: Mamba enough for temporal compression and teacher forcing enough for training. No new physical entities are proposed.

free parameters (5)
  • Foreground threshold = 0.5
    Used in filter_fg during unprompted inference; fixed at 0.5 across all benchmarks after an ablation showed robustness over 0.3-0.7. Though not tuned per dataset, it is a manually chosen number affecting results.
  • Number of object queries (N_det) and ID vectors (N_id) = 100 each
    Set in Appendix A; capacity for detected objects, affects detection and tracking ability.
  • History Compressor layers = 6
    Depth of temporal/spatial fusion module; chosen by hand.
  • History Decoder layers = 6
    Depth of cross-attention decoder; chosen by hand.
  • Feature stride = 8
    Uses 1/8 resolution feature map from Swin backbone; this is an architectural choice that trades detail for memory. The limitations section mentions this as a reason for the VOS gap.
axioms (5)
  • standard math The factorization P(y_t | y0, y<t, I<=t) is a valid probabilistic decomposition of the video segmentation problem.
    Used to justify the autoregressive framing in Sec 2.1, Eq (2). Assumes appropriate conditional independence and existence of densities, standard in sequence models.
  • domain assumption A fixed-size Mamba state can represent all relevant past spatio-temporal information for segmentation.
    Core assumption of History Compressor (Sec 2.2); if false, long-video segmentation fails.
  • domain assumption Ground-truth masks and labels in the training datasets are accurate and cover all objects of interest.
    Used throughout training (Sec 3.2) and evaluation; any label noise affects results.
  • domain assumption Teacher forcing with randomly sampled t_sample generalizes to inference with autoregressive feedback.
    Training pipeline (Sec 2.3) uses ground-truth history; inference (Alg 1) uses predicted masks. Assumes exposure bias is manageable.
  • domain assumption The Token Mark operation preserves mask details better than vectorized instance compression.
    Invoked in History Marker (Sec 2.2) and justified by prior work (Heo et al. 2025), not proven in this paper.
invented entities (2)
  • History Compressor module no independent evidence
    purpose: Compresses all past frame and mask information into a single fixed-size spatial state using Mamba and self-attention for constant-memory long-video inference.
    Introduced in this paper; its benefit is shown only through in-paper ablations, no external validation.
  • ID vectors (A) and mask buffer (M) no independent evidence
    purpose: Track detected instances across frames by maintaining one-to-one mapping between allocated vectors and previous mask predictions.
    Internal bookkeeping mechanism; no external evidence of necessity beyond the paper's experiments.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Autoregressive Universal Video Segmentation Model." pith.science (2026). https://pith.science/paper/RVGJYAVY

@misc{pith2026250819242,
  author       = {Pith},
  title        = {Pith review of: Autoregressive Universal Video Segmentation Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RVGJYAVY}},
  note         = {Machine review of arXiv:2508.19242}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent video foundation models such as SAM2 excel at prompted video segmentation by treating masks as a general-purpose primitive. However, many real-world settings require unprompted segmentation that aims to detect and track all objects in a video without external cues, leaving today's landscape fragmented across task-specific models and pipelines. We recast streaming video segmentation as sequential mask prediction, analogous to language modeling, and introduce the Autoregressive Universal Segmentation Model (AUSM), a single architecture that unifies both prompted and unprompted video segmentation. Built on recent state-space models, AUSM maintains a fixed-size spatial state and scales to video streams of arbitrary length. Furthermore, all components of AUSM are designed for parallel training across frames, yielding substantial speedups over iterative training. On standard benchmarks (DAVIS17, YouTube-VOS 2018 & 2019, MOSE, YouTube-VIS 2019 & 2021, and OVIS) AUSM outperforms prior universal streaming video segmentation methods and achieves up to 2.5x faster training on 16-frame sequences.

Figures

Figures reproduced from arXiv: 2508.19242 by Albert Gu, Min-Hung Chen, Miran Heo, Ryo Hachiuma, Seon Joo Kim, Sukjun Hwang, Yu-Chiang Frank Wang.

Figure 1
Figure 1. Figure 1: High-level overview of AUSM during training and inference. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: HistoryCompressor mod￾ule. Mamba encodes temporal de￾pendencies, while self-attention cap￾tures spatial structure, enabling recur￾rent compression of 𝐸 with constant memory. Specifically, the temporal mamba layer operates pixel-wise, mix￾ing information of each pixel through￾out the time dimension 𝑇. The spa￾tial self-attention layer, by contrast, is frame-independent that fuses informa￾tion spanning over … view at source ↗
Figure 3
Figure 3. Figure 3: Schematic of Preprocess. The video contains three ground-truth instances: person1 (the person riding the horse), person2 (the person standing), and horse. Each instance is matched with a vector from 𝒜, represented by colored circles: red for person1, green for person2, and blue for horse. The highlighted contours indicate the randomly sampled timesteps (𝑡 𝑖 sample) for each instance. As illustrated in [PI… view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison between Stage 2 (5-frame) and Stage 3 (16-frame) training across four benchmark datasets. task-specific fine-tuning. We report AUSM with Swin-T and Swin-B backbones (Liu et al., 2021). Prompted Video Segmentation. A leading specialized method for prompted video segmentation is SAM2 (Ravi et al., 2025), a memory-based mask-propagation approach built on STM (Oh et al., 2019), where eac… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison between unprompted and prompted video segmentation results using a single [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison between unprompted and prompted video segmentation results using a single [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 44 canonical work pages

  1. [1]

    Just read twice: closing the recall gap for recurrent language models

    Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher Ré. Just read twice: closing the recall gap for recurrent language models. arXiv preprint arXiv:2407.05483, 2024. 9

    Pith/arXiv arXiv 2024

  2. [2]

    Tarvis: A unified approach for target-based video segmentation

    Ali Athar, Alexander Hermans, Jonathon Luiten, Deva Ramanan, and Bastian Leibe. Tarvis: A unified approach for target-based video segmentation. InCVPR, 2023. 1, 7, 10

    work page 2023

  3. [3]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. InECCV, 2020. 5, 6, 10

    work page 2020

  4. [4]

    Per-pixel classification is not all you need for semantic segmentation

    Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. InNeurIPS, 2021. 5

    work page 2021

  5. [5]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. InCVPR, 2022. 5, 10, 12

    work page 2022

  6. [6]

    Xmem: Long-term video object segmentation with an atkinson- shiffrin memory model

    Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an atkinson- shiffrin memory model. InECCV, 2022. 7, 10

    work page 2022

  7. [7]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benen- son, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InCVPR, 2016. 2

    work page 2016

  8. [8]

    Mose: A new dataset for video object segmentation in complex scenes

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. Mose: A new dataset for video object segmentation in complex scenes. InICCV, 2023. 2, 6, 7

    work page 2023

  9. [9]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. 3, 5

    Pith/arXiv arXiv 2024

  10. [10]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023. 2, 4, 9, 10

    Pith/arXiv arXiv 2023

  11. [11]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. 10

    Pith/arXiv arXiv 2021

  12. [12]

    On the parameterization and initialization of diagonal state space models

    Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. InNeurIPS, 2022. 10

    work page 2022

  13. [13]

    Vita: Video instance segmentation via object token association

    Miran Heo, Sukjun Hwang, Seoung Wug Oh, Joon-Young Lee, and Seon Joo Kim. Vita: Video instance segmentation via object token association. InNeurIPS, 2022. 4, 7, 10

    work page 2022

  14. [14]

    A generalized framework for video instance segmentation

    Miran Heo, Sukjun Hwang, Jeongseok Hyun, Hanjung Kim, Seoung Wug Oh, Joon-Young Lee, and Seon Joo Kim. A generalized framework for video instance segmentation. InCVPR, 2023. 4, 5, 7, 8, 9

    work page 2023

  15. [15]

    Omni-rgpt: Unifyingimageandvideoregion-levelunderstanding via token marks

    Miran Heo, Min-Hung Chen, De-An Huang, Sifei Liu, Subhashree Radhakrishnan, Seon Joo Kim, Yu- ChiangFrankWang, andRyoHachiuma. Omni-rgpt: Unifyingimageandvideoregion-levelunderstanding via token marks. InCVPR, 2025. 2, 4

    work page 2025

  16. [16]

    Robust and consistent online video instance segmentation via instance mask propagation

    Miran Heo, Seoung Wug Oh, Seon Joo Kim, and Joon-Young Lee. Robust and consistent online video instance segmentation via instance mask propagation. InAAAI, 2025. 2, 4, 5, 9

    work page 2025

  17. [17]

    Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024. 11 16 Autoregressive Universal Video Segmentation Model

    Pith/arXiv arXiv 2024

  18. [18]

    Minvis: A minimal video instance segmentation framework without video-based training

    De-An Huang, Zhiding Yu, and Anima Anandkumar. Minvis: A minimal video instance segmentation framework without video-based training. InNeurIPS, 2022. 1, 9

    work page 2022

  19. [19]

    Video instance segmentation using inter-frame communication transformers

    Sukjun Hwang, Miran Heo, Seoung Wug Oh, and Seon Joo Kim. Video instance segmentation using inter-frame communication transformers. InNeurIPS, 2021. 5, 10

    work page 2021

  20. [20]

    Video panoptic segmentation

    Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Video panoptic segmentation. InCVPR,

    work page

  21. [21]

    Tubeformer-deeplab: Video mask transformer

    Dahun Kim, Jun Xie, Huiyu Wang, Siyuan Qiao, Qihang Yu, Hong-Seok Kim, Hartwig Adam, In So Kweon, and Liang-Chieh Chen. Tubeformer-deeplab: Video mask transformer. InCVPR, 2022. 10

    work page 2022

  22. [22]

    Visage: Video instance segmentation with appearance-guided enhancement

    Hanjung Kim, Jaehyun Kang, Miran Heo, Sukjun Hwang, Seoung Wug Oh, and Seon Joo Kim. Visage: Video instance segmentation with appearance-guided enhancement. InECCV, 2024. 1, 7, 8, 9

    work page 2024

  23. [23]

    Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In ICCV, 2023. 2, 10

    work page 2023

  24. [24]

    Univs: Unified and universal video segmentation with prompts as queries

    Minghan Li, Shuai Li, Xindong Zhang, and Lei Zhang. Univs: Unified and universal video segmentation with prompts as queries. InCVPR, 2024. 1, 2, 4, 7, 8, 10

    work page 2024

  25. [25]

    Video k-net: A simple, strong, and unified baseline for video segmentation

    XiangtaiLi, WenweiZhang, JiangmiaoPang, KaiChen, GuangliangCheng, YunhaiTong, andChenChange Loy. Video k-net: A simple, strong, and unified baseline for video segmentation. InCVPR, 2022. 7, 10

    work page 2022

  26. [26]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InECCV, 2014. 6, 7, 12

    work page 2014

  27. [27]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InICCV, 2021. 8, 12

    work page 2021

  28. [28]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2019. 12

    2019

  29. [29]

    Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 1:3, 2020

    Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 1:3, 2020. 3, 5

    Pith/arXiv arXiv 2005

  30. [30]

    Trackformer: Multi- object tracking with transformers

    Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multi- object tracking with transformers. InCVPR, 2022. 5

    work page 2022

  31. [31]

    Mot16: A benchmark for multi-object tracking.arXiv preprint arXiv:1603.00831, 2016

    Anton Milan, Laura Leal-Taixé, Ian Reid, Stefan Roth, and Konrad Schindler. Mot16: A benchmark for multi-object tracking.arXiv preprint arXiv:1603.00831, 2016. 11

    Pith/arXiv arXiv 2016

  32. [32]

    Videoobjectsegmentationusingspace-time memory networks

    SeoungWugOh, Joon-YoungLee, NingXu, andSeonJooKim. Videoobjectsegmentationusingspace-time memory networks. InICCV, 2019. 2, 4, 8, 10

    work page 2019

  33. [33]

    The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017. 2, 3, 6, 7

    Pith/arXiv arXiv 2017

  34. [34]

    Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409, 2021

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409, 2021. 11

    Pith/arXiv arXiv 2021

  35. [35]

    Occluded video instance segmentation: A benchmark.IJCV, 2022

    Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation: A benchmark.IJCV, 2022. 2, 6, 7, 12

    work page 2022

  36. [36]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019. 3 17 Autoregressive Universal Video Segmentation Model

    work page 2019

  37. [37]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, 2021. 12

    work page 2021

  38. [38]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. InICLR, 2025. 2, 6, 7, 8, 10

    work page 2025

  39. [39]

    Urvos: Unified referring video object segmentation network with a large-scale benchmark

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. InECCV, 2020. 11

    work page 2020

  40. [40]

    Repetition improves language model embeddings.arXiv preprint arXiv:2402.15449, 2024

    Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings.arXiv preprint arXiv:2402.15449, 2024. 9

    Pith/arXiv arXiv 2024

  41. [41]

    Sequence to sequence learning with neural networks

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NeurIPS, 2014. 5

    work page 2014

  42. [42]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, 2017. 2, 4, 9, 10

    work page 2017

  43. [43]

    Mots: Multi-object tracking and segmentation

    PaulVoigtlaender, MichaelKrause, AljosaOsep, JonathonLuiten, BerinBalachandarGnanaSekar, Andreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmentation. InCVPR, 2019. 2

    work page 2019

  44. [44]

    Max-deeplab: End-to-end panoptic segmentation with mask transformers

    Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. InCVPR, 2021. 10

    work page 2021

  45. [45]

    End-to-end video instance segmentation with transformers

    Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. InCVPR, 2020. 10

    work page 2020

  46. [46]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. InNeurIPS, 2022. 9

    work page 2022

  47. [47]

    Segment every reference object in spatial and temporal spaces

    Jiannan Wu, Yi Jiang, Bin Yan, Huchuan Lu, Zehuan Yuan, and Ping Luo. Segment every reference object in spatial and temporal spaces. InICCV, 2023. 10

    work page 2023

  48. [48]

    Uniref++: Segment every reference object in spatial and temporal spaces.arXiv preprint arXiv:2312.15715, 2023

    Jiannan Wu, Yi Jiang, Bin Yan, Huchuan Lu, Zehuan Yuan, and Ping Luo. Uniref++: Segment every reference object in spatial and temporal spaces.arXiv preprint arXiv:2312.15715, 2023. 7, 10

    Pith/arXiv arXiv 2023

  49. [49]

    In defense of online models for video instance segmentation

    Junfeng Wu, Qihao Liu, Yi Jiang, Song Bai, Alan Yuille, and Xiang Bai. In defense of online models for video instance segmentation. InECCV, 2022. 1, 9, 10

    work page 2022

  50. [50]

    Online object tracking: A benchmark

    Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Online object tracking: A benchmark. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 2411–2418, 2013. 11

    work page 2013

  51. [51]

    Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023. 11

    Pith/arXiv arXiv 2023

  52. [52]

    Youtube-vos: A large-scale video object segmentation benchmark.arXiv preprint arXiv:1809.03327, 2018

    Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark.arXiv preprint arXiv:1809.03327, 2018. 2, 6

    Pith/arXiv arXiv 2018

  53. [53]

    Universal instance perception as object discovery and retrieval

    Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. InCVPR, 2023. 1, 2, 7, 10

    work page 2023

  54. [54]

    Video instance segmentation

    Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. InICCV, 2019. 1, 2, 3, 6, 7, 12 18 Autoregressive Universal Video Segmentation Model

    work page 2019

  55. [55]

    Decoupling features in hierarchical propagation for video object segmentation

    Zongxin Yang and Yi Yang. Decoupling features in hierarchical propagation for video object segmentation. In NeurIPS, 2022. 7, 10

    work page 2022

  56. [56]

    Associating objects with transformers for video object segmen- tation

    Zongxin Yang, Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmen- tation. InNeurIPS, 2021. 2, 4, 10

    work page 2021

  57. [57]

    Ctvis: Consistent training for online video instance segmentation

    Kaining Ying, Qing Zhong, Weian Mao, Zhenhua Wang, Hao Chen, Lin Yuanbo Wu, Yifan Liu, Chengxiang Fan, Yunzhi Zhuge, and Chunhua Shen. Ctvis: Consistent training for online video instance segmentation. In ICCV, 2023. 1, 9

    work page 2023

  58. [58]

    Dvis: Decoupled video instance segmentation framework

    Tao Zhang, Xingye Tian, Yu Wu, Shunping Ji, Xuebo Wang, Yuan Zhang, and Pengfei Wan. Dvis: Decoupled video instance segmentation framework. InICCV, 2023. 7 19

    work page 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.