Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Towards Real-Time Open-Vocabulary Video Instance Segmentation

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TROY-VIS segments arbitrary objects in video at 25 FPS, 20x faster than the prior best, with equal or better accuracy.

desk verdict A capable engineering contribution to real-time OV-VIS, but the paper's own FPS numbers contradict each other and the lower one misses its real-time bar. read the letter →

arxiv 2412.04434 v1 pith:U33OJMH5 submitted 2024-12-05 cs.CV

Bin Yan , Martin Sundermeyer , David Joseph Tan , Huchuan Lu , Federico Tombari This is my paper · ORCID
classification cs.CV
keywords open-vocabularyvideoinstancesegmentationreal-timeinferencekernelinterpolationflashembeddingmemorydecoupledattentiontemporalconsistencyDETR-styledecoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-vocabulary video instance segmentation (OV-VIS) has been accurate but too slow for real-world use: the lightweight GLEE-Lite foundation model processes about 1.25 frames per second on an A100 GPU. This paper argues that OV-VIS can run in real time without giving up accuracy, and introduces TROY-VIS, a model that runs at 25 FPS on the same hardware while matching or exceeding GLEE-Lite on the BURST and LV-VIS benchmarks. The route is to attack three computational bottlenecks: cross-modal attention, repeated text-embedding computation, and the per-frame instance decoder. If the claim holds, the practical obstacle to deploying open-vocabulary video understanding in robotics and augmented reality is removed.

What carries the argument

The argument is carried by three mechanisms plus the instance-kernel representation they serve. Instance kernels are per-object weight arrays that are convolved with a downsampled pixel embedding map to produce masks, and the expensive part is producing those kernels with a 9-layer transformer decoder. The Decoupled Attention Feature Enhancer splits the original modality-scale hybrid cross-attention into a modality-attention between text and the lowest-resolution visual tokens and a scale-attention supplied by the existing deformable encoder, which relates points across feature scales, cutting the attention cost by a factor of roughly 85. Flash Embedding Memory stores category-name to text-embedding pairs so retrieval is O(1) instead of re-running the text encoder every frame, which also makes a stronger EVA-02-CLIP-L text encoder affordable. Kernel Interpolation assumes temporal consistency between frames: accurate kernels are computed on key frames spaced F=3 frames apart, and non-key frames use proxy kernels obtained by linear, nearest-neighbor, or causal nearest-neighbor interpolation; the causal variant keeps this compatible with online applications. The full model uses an EfficientViT-L2 backbone and only 3 encoder and 3 decoder layers.

What would settle it

Take a video benchmark with fast-moving or frequently occluded objects, run TROY-VIS with key frames spaced three frames apart, and compare tracking accuracy (HOTA or mask IoU) against running the instance decoder on every frame. If the gap in accuracy is much larger than the paper's reported 0.4 AP drop on LV-VIS, the temporal-consistency assumption is the limiting factor and the claim that real-time speed costs no accuracy would not generalize.

Watch

Extended reading notes

Core claim

TROY-VIS is presented as the first real-time open-vocabulary video instance segmentation model. On LV-VIS it reaches 20.9 AP overall, 23.4 AP on base categories and 19.1 AP on novel categories, while GLEE-Lite reaches 19.6, 22.1 and 17.7; on BURST it reaches HOTA 23.9 on all categories and 42.3 on common categories, against 22.6 and 36.4 for GLEE-Lite. The speed gain to 40 ms per frame (25 FPS) comes from three changes: decoupling the feature enhancer's modality and scale attention, caching text embeddings in a Flash Embedding Memory so categories are encoded once, and interpolating instance kernels between key frames so the heavy decoder runs only one frame in three. The paper reports that these changes cut per-frame latency from 805 ms to 40 ms with no net loss of accuracy.

Load-bearing premise

The load-bearing premise is that objects change little between key frames spaced three frames apart, so kernels computed on an earlier frame still match the current frame; fast motion, occlusion, or appearance change inside that gap would make the reused predictions stale.

Editorial extensions

If this is right

  • Real-time open-vocabulary video understanding becomes possible on a single A100 GPU, the paper's stated enabler for mobile robotics and augmented reality.
  • The text-encoder bottleneck disappears on fixed-vocabulary benchmarks: category embeddings are computed once and reused, so larger text encoders can be swapped in at no inference cost.
  • Because kernel interpolation is applied only at inference, the model can be trained primarily on image-level annotations and does not require video data for the interpolation itself.
  • On both evaluated benchmarks, TROY-VIS sets the best accuracy numbers among efficient OV-VIS methods while being an order of magnitude faster than GLEE-Lite.
  • Reducing the decoder from 9 to 3 layers is shown to trade 1.2 LV-VIS AP for a 35 ms latency saving, and kernel interpolation adds a further 0.4 AP drop while raising speed from 22.2 to 31.3 FPS.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper does not run is varying the key-frame interval F: if F=3 already relies on temporal continuity, then F=1 versus F=3 on fast-motion videos would expose how much of the speed gain depends on static scenes.
  • The three interpolation variants are described but the final model's choice is not ablated; comparing linear, nearest-neighbor, and causal nearest-neighbor on the same benchmark would clarify which strategy the reported gains come from.
  • Flash Embedding Memory's nearest-neighbor fallback for unseen categories is likely to work better for categories near the existing vocabulary; rare or abstract categories may need a different retrieval rule, an extension the paper leaves implicit.
  • The same decoupled-attention and kernel-interpolation pattern could transfer to other DETR-style video perception tasks, such as open-vocabulary detection or panoptic segmentation, though the paper does not claim this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes TROY-VIS, a video instance segmentation architecture designed for open-vocabulary video instance segmentation (OV-VIS) at real-time speed. It identifies three computational bottlenecks in the GLEE-Lite baseline—text encoder, feature enhancer, and instance decoder—and introduces three corresponding techniques: a decoupled attention feature enhancer, a Flash Embedding Memory for caching text embeddings, and a kernel interpolation mechanism that propagates instance kernels from key frames to non-key frames. The authors report that TROY-VIS runs at 25 FPS on an A100 GPU, about 20x faster than GLEE-Lite, while achieving competitive or better accuracy on the BURST and LV-VIS benchmarks. The experiments include component-wise latency analysis (Table 1), zero-shot comparison with prior methods (Table 2), and an incremental ablation study (Table 3).

Significance. If the reported speed and accuracy numbers hold, this would be a practically valuable contribution: the first open-vocabulary video instance segmentation model that runs in real time, with strong zero-shot accuracy on two large-vocabulary benchmarks. The paper benefits from a clear component-wise latency breakdown, an incremental ablation design, and zero-shot evaluation on two external benchmarks, which makes the empirical claims easy to inspect. However, the central 'real-time' claim currently rests on internally inconsistent FPS numbers: the abstract and Table 1 imply 25 FPS, while Table 2 reports 20.9 FPS, which is below the paper's own 24 FPS real-time threshold. This inconsistency must be resolved before the headline claim can be accepted.

major comments (3)
  1. [Abstract; §4.3, Table 2; §1, Table 1] The FPS figures for the final model are mutually inconsistent. The abstract and the efficiency paragraph in Section 4.3 state 25 FPS (40 ms per frame), while Table 2 lists TROY-VIS at 20.9 FPS (47.8 ms per frame). Since Section 1 defines real-time as 24 FPS, the two numbers fall on opposite sides of the threshold. The manuscript does not explain whether the Table 2 FPS is an end-to-end measurement that includes text embedding retrieval, kernel interpolation, and post-processing, or whether the 40 ms in Table 1 is a component-only forward pass. This is load-bearing because the headline 'first real-time OV-VIS model' and the '20x faster than GLEE-Lite' claim are both direct consequences of the speed number; the authors need to report one consistent, clearly defined measurement protocol.
  2. [§3.3, Table 3] The interpolation variant used in the final model is never identified. Section 3.3 presents three options: linear, nearest neighbor, and causal nearest neighbor. Neither Table 2 nor Table 3 states which one is used in the final TROY-VIS model. This matters because linear and bi-directional nearest-neighbor interpolation access future key frames, which is incompatible with the claimed online/real-time applicability; only causal nearest neighbor supports that claim. The paper also provides no ablation over the interpolation variant or over the key-frame interval F=3, even though F directly controls the speed/accuracy trade-off of the kernel interpolation mechanism.
  3. [§3.3, Eq. (1)] Kernel interpolation assumes that elements of K(T) and K(T+3) correspond to the same object instances, but the paper does not describe how this correspondence is established. The instance decoder runs independently on each key frame with N unordered object queries; without a matching step, linearly combining kernel vectors across key frames mixes different instances, and even causal nearest-neighbor copying can attach a stale kernel to the wrong object after a reordering of queries. The authors need to specify the matching mechanism, or show experimentally that query order is stable enough for direct interpolation.
minor comments (4)
  1. [§4.4, Table 3] The final row of Table 3 reports AP=15.7 under the lightweight 100K-iteration training setting, whereas Table 2 reports AP=20.9 for TROY-VIS under the full 300K-iteration setting described in Section 3.4. The text should state this difference explicitly so that readers do not directly compare the two tables as if they reflect the same model.
  2. [§3.2] For unseen categories encountered in the wild, the paper proposes averaging the embeddings of K nearest neighbors from the memory. This heuristic is plausible but is not validated in any experiment; a sentence noting this limitation, or a small analysis, would improve the presentation.
  3. [Abstract; §4.1] The paper says the code and model will be released, but no code is currently available. Given that the speed claims depend on exact measurement conditions, the authors should release timing scripts or a precise measurement protocol at revision time.
  4. [Figure 1] Figure 1 states that 'TROY-VIS is the only method that runs in real-time.' This statement is only true if the FPS value is the 25 FPS figure from the abstract and Table 1; if Table 2's 20.9 FPS is the correct end-to-end value, this claim would be false under the paper's own 24 FPS definition. The figure and table should be made consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's accuracy and speed claims are empirical benchmark results with external baselines and component ablations.

full rationale

This is an empirical systems paper, and its central claims do not reduce to their own inputs. The main accuracy results (LV-VIS AP 20.9, BURST HOTA 23.9) are measured on external benchmarks, BURST and LV-VIS, against published baselines such as GLEE-Lite; they are not derived from the method's definitions or from fitted parameters. Each introduced component is evaluated through incremental ablations in Table 3, where latency and AP are measured rather than constructed: adding decoupled attention, Flash Embedding Memory, reduced encoder/decoder layers, kernel interpolation, and EfficientViT-L2 changes the measured numbers, and no component is tuned to reproduce a target result. Flash Embedding Memory caches embeddings from a frozen external text encoder (EVA-02-CLIP-L); its inference behavior is a retrieval operation, not a prediction derived from the method's own output. Kernel interpolation is an explicit inductive bias based on temporal consistency, and the paper clearly states it produces 'proxy kernels' rather than claiming these are independently derived; even if this assumption fails on fast motion, that is a robustness or correctness concern, not circularity. The only prior works by the authors cited in the paper are Unicorn [44] and UNINEXT [45], and these are used as related work, not as load-bearing justification for TROY-VIS's design choices; the baseline GLEE [40] is external. There is no uniqueness theorem imported from the authors, no ansatz smuggled in via self-citation, and no renaming of a known result. The skeptical observation that the paper reports contradictory FPS figures (25 FPS versus Table 2's 20.9 FPS) is an internal consistency and measurement-reporting concern; it does not make any claim equivalent to its inputs by construction. Overall, the derivation chain is self-contained with respect to the benchmark evaluations and component ablations, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on empirical engineering choices (key-frame interval F=3, decoder depth 3) and on the domain assumption that video frames are temporally consistent. No invented physical or conceptual entities are introduced.

free parameters (4)
  • Key-frame interval F = 3
    Set in Sec. 3.3 ('F = 3 in this work') without ablating other values. This determines how often the full instance decoder runs and is central to the speed/accuracy trade-off.
  • Instance decoder layers = 3
    Sec. 3.3 reduces the decoder from 9 layers to 3. Ablation #4 bundles encoder and decoder reduction together, so the individual effect of decoder depth is not isolated.
  • Deformable encoder layers = 3
    Sec. 3.1 reduces the deformable encoder from 6 layers to 3. Ablation treats encoder depth jointly with decoder depth.
  • K in Flash Embedding Memory nearest-neighbor retrieval = not specified
    Sec. 3.2 describes embedding unseen categories by averaging the K nearest neighbors in the memory, but K and the similarity metric are never specified, so the in-the-wild branch is not reproducible.
assumptions (3)
  • domain assumption Adjacent video frames are temporally consistent; object instance kernels change slowly over the F=3 frame interval.
    Premise of kernel interpolation, stated in Sec. 3.3. If objects move quickly, get occluded, or change appearance within three frames, proxy kernels from key frames become inaccurate and tracking may fail.
  • domain assumption Text embeddings for a fixed category name are constant across frames and can be cached without loss.
    Basis of Flash Embedding Memory (Sec. 3.2). This is true for a frozen text encoder; the paper fine-tunes the text encoder during training but caches at inference.
  • domain assumption EfficientViT-L2 provides a better accuracy/speed trade-off than ResNet-50 as the vision backbone.
    Adopted from prior work [5]. The paper's ablation #6 shows an AP gain of 1.0 over #5 at an 8 ms latency increase, which supports but does not fully prove that the trade-off is beneficial in all deployment scenarios.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Real-Time Open-Vocabulary Video Instance Segmentation." pith.science (2026). https://pith.science/paper/U33OJMH5

@misc{pith2026241204434,
  author       = {Pith},
  title        = {Pith review of: Towards Real-Time Open-Vocabulary Video Instance Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U33OJMH5}},
  note         = {Machine review of arXiv:2412.04434}
}
read the original abstract

In this paper, we address the challenge of performing open-vocabulary video instance segmentation (OV-VIS) in real-time. We analyze the computational bottlenecks of state-of-the-art foundation models that performs OV-VIS, and propose a new method, TROY-VIS, that significantly improves processing speed while maintaining high accuracy. We introduce three key techniques: (1) Decoupled Attention Feature Enhancer to speed up information interaction between different modalities and scales; (2) Flash Embedding Memory for obtaining fast text embeddings of object categories; and, (3) Kernel Interpolation for exploiting the temporal continuity in videos. Our experiments demonstrate that TROY-VIS achieves the best trade-off between accuracy and speed on two large-scale OV-VIS benchmarks, BURST and LV-VIS, running 20x faster than GLEE-Lite (25 FPS v.s. 1.25 FPS) with comparable or even better accuracy. These results demonstrate TROY-VIS's potential for real-time applications in dynamic environments such as mobile robotics and augmented reality. Code and model will be released at https://github.com/google-research/troyvis.

Figures

Figures reproduced from arXiv: 2412.04434 by the authors.

Figure 1
Figure 1. Performance and speed comparison on the LV-VIS [ [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Architecture comparison between the original feature enhancer and our decoupled attention feature enhancer. In our design, a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of kernel interpolation. Cuboids represent [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative results of TROY-VIS on challenging indoor and outdoor scenarios. Best viewed in color with zoom-in. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OpenFusion++: An Open-vocabulary Real-time Scene Understanding System

    cs.CV 2025-04 conditional novelty 4.0 of 10

    OpenFusion++ upgrades the OpenFusion real-time 3D mapping system with confidence-based boundary refinement, area-weighted semantic caching, and a two-stage query matching that improves semantic accuracy and spatial qu...

Reference graph

Works this paper leans on

61 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Tarvis: A unified approach for target-based video segmentation

    Ali Athar, Alexander Hermans, Jonathon Luiten, Deva Ra- manan, and Bastian Leibe. Tarvis: A unified approach for target-based video segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18738–18748, 2023. 2

    work page 2023

  3. [3]

    Burst: A benchmark for unifying object recognition, segmentation and tracking in video

    Ali Athar, Jonathon Luiten, Paul V oigtlaender, Tarasha Khu- rana, Achal Dave, Bastian Leibe, and Deva Ramanan. Burst: A benchmark for unifying object recognition, segmentation and tracking in video. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision , pages 1674–1683, 2023. 1, 2, 3, 7, 8

    work page 2023

  4. [4]

    Simple online and realtime tracking

    Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP) , pages 3464–3468. IEEE, 2016. 2, 8

    work page 2016

  5. [5]

    Efficientvit: Lightweight multi-scale attention for high- resolution dense prediction

    Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi-scale attention for high- resolution dense prediction. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 17302– 17313, 2023. 3, 6, 7

    work page 2023

  6. [6]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In ECCV, 2020. 3

    2020

  7. [7]

    Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model

    Ho Kei Cheng and Alexander G Schwing. Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model. In European Conference on Computer Vi- sion, pages 640–658. Springer, 2022. 2, 8

    work page 2022

  8. [8]

    Yolo-world: Real-time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xing- gang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024. 2

    work page 2024

Show all 61 references
  1. [9]

    Xception: Deep learning with depthwise separable convolutions

    Franc ¸ois Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 1251–1258, 2017. 4

    2017

  2. [10]

    Tao: A large-scale benchmark for tracking any object

    Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. Tao: A large-scale benchmark for tracking any object. In ECCV, 2020. 7

    2020

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 2

    2018 arXiv

  4. [12]

    Eva-02: A visual representation for neon genesis

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xin- long Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303.11331, 2023. 2, 5, 8

    2023 arXiv

  5. [13]

    Ultralyt- ics yolov8

    Ayush Chaurasia Glenn Jocher and Jing Qiu. Ultralyt- ics yolov8. https://github.com/ultralytics/ ultralytics, 2023. 2

    2023

  6. [14]

    Lvis: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019. 6

    2019

  7. [15]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 3

    2022

  8. [16]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,

  9. [17]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  10. [18]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, 2023. 1, 2, 3, 6

    2023

  11. [19]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of compute...

    2017

  12. [20]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and vi...

    2020

  13. [21]

    Mask dino: Towards a unified transformer-based framework for object detection and segmentation

    Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Yeung Shum. Mask dino: Towards a unified transformer-based framework for object detection and segmentation. In CVPR, 2023. 2, 3

    2023

  14. [22]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In CVPR, pages 10965–10975,

  15. [23]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 6

    2014

  16. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 6

    2019

  17. [25]

    Hota: A higher order metric for evaluating multi-object tracking

    Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taix´e, and Bastian Leibe. Hota: A higher order metric for evaluating multi-object tracking. International journal of computer vision, 129:548– 578, 2021. 7

    2021

  18. [26]

    Simple open-vocabulary object detection with vision transformers

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection with vision transformers. In European Conference on Computer...

    2022

  19. [27]

    Mod- eling context between objects for referring expression under- standing

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Mod- eling context between objects for referring expression under- standing. In ECCV, 2016. 6

    2016

  20. [28]

    OpenAI. Sora. https : / / openai . com / index / video - generation - models - as - world - simulators/, 2024. 2

    2024

  21. [29]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 6

    2023 arXiv

  22. [30]

    Occluded video instance segmentation: A bench- mark

    Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation: A bench- mark. International Journal of Computer Vision, pages 1–18,

  23. [31]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  24. [32]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 1

    2024 arXiv

  25. [33]

    Grounding dino 1.5: Advance the” edge” of open-set object detection

    Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wen- long Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al. Grounding dino 1.5: Advance the” edge” of open-set object detection. arXiv preprint arXiv:2405.10300, 2024. 2, 3

    2024 arXiv

  26. [34]

    Urvos: Unified referring video object segmentation network with a large-scale benchmark

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16 , pages 208–223. Springer,

    2020

  27. [35]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 8430–8439, 2019. 6

    2019

  28. [36]

    Aligning and prompting everything all at once for univer- sal visual perception

    Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for univer- sal visual perception. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogn...

    2024

  29. [37]

    Mobile- clip: Fast image-text models through multi-modal reinforced training

    Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobile- clip: Fast image-text models through multi-modal reinforced training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15963–...

    2024

  30. [38]

    Towards open-vocabulary video instance segmentation

    Haochen Wang, Cilin Yan, Shuai Wang, Xiaolong Jiang, Xu Tang, Yao Hu, Weidi Xie, and Efstratios Gavves. Towards open-vocabulary video instance segmentation. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 4057–4066, 2023. 1, 2, 3, 7, 8

    2023

  31. [39]

    Unidentified video objects: A benchmark for dense, open- world segmentation

    Weiyao Wang, Matt Feiszli, Heng Wang, and Du Tran. Unidentified video objects: A benchmark for dense, open- world segmentation. In ICCV, 2021. 6

    2021

  32. [40]

    General object foundation model for images and videos at scale

    Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. General object foundation model for images and videos at scale. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 3783–3795, 2024. 1, 2, 3, 6, 7, 8

    2024

  33. [41]

    Tinyclip: Clip dis- tillation via affinity mimicking and weight inheritance

    Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang, et al. Tinyclip: Clip dis- tillation via affinity mimicking and weight inheritance. In Proceedings of the IEEE/CVF International Conference on C...

    2023

  34. [42]

    Efficientsam: Leveraged masked image pretraining for efficient segment anything

    Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xi- ang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. In Proceedings of the IEEE/CVF Conference on Compu...

    2024

  35. [43]

    Ning Xu, Linjie Yang, Jianchao Yang, Dingcheng Yue, Yuchen Fan, Yuchen Liang, and Thomas S. Huang. Youtube- vis dataset 2021 version. https://youtube- vos. org/dataset/vis/. 1, 3, 6

    2021

  36. [44]

    Towards grand unification of object tracking

    Bin Yan, Yi Jiang, Peize Sun, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu. Towards grand unification of object tracking. In ECCV, 2022. 2

    2022

  37. [45]

    Universal instance perception as object discovery and retrieval

    Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Ze- huan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In CVPR, 2023. 2, 3

    2023

  38. [46]

    Video instance seg- mentation

    Linjie Yang, Yuchen Fan, and Ning Xu. Video instance seg- mentation. In ICCV, 2019. 1, 3, 6

    2019

  39. [47]

    Detclip: Dictionary-enriched visual-concept paralleled pre- training for open-world detection

    Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre- training for open-world detection. In NeurIPS, 2022. 3

    2022

  40. [48]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In ECCV, 2016. 6

    2016

  41. [49]

    Open-vocabulary object detection using captions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih- Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14393–14402, 2021. 2

    2021

  42. [50]

    Faster segment anything: Towards lightweight sam for mo- bile applications

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mo- bile applications. arXiv preprint arXiv:2306.14289 , 2023. 3

    2023 arXiv

  43. [51]

    Dino: Detr with improved denoising anchor boxes for end-to-end object detection

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. 2023. 2, 3

    2023

  44. [52]

    A simple framework for open-vocabulary segmentation and detection

    Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 1020–1031, 2023. 2

    2023

  45. [53]

    Mobileinst: Video in- stance segmentation on the mobile

    Renhong Zhang, Tianheng Cheng, Shusheng Yang, Haoyi Jiang, Shuai Zhang, Jiancheng Lyu, Xin Li, Xiaowen Ying, Dashan Gao, Wenyu Liu, et al. Mobileinst: Video in- stance segmentation on the mobile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, page...

    2024

  46. [54]

    Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection

    Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li. Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 9759–9768, 2020. 2

    2020

  47. [55]

    Efficientvit-sam: Accelerated segment anything model without performance loss

    Zhuoyang Zhang, Han Cai, and Song Han. Efficientvit-sam: Accelerated segment anything model without performance loss. arXiv preprint arXiv:2402.05008, 2024. 3

    2024 arXiv

  48. [56]

    Real-time transformer-based open-vocabulary detection with efficient fusion head

    Tiancheng Zhao, Peng Liu, Xuan He, Lu Zhang, and Kyu- song Lee. Real-time transformer-based open-vocabulary detection with efficient fusion head. arXiv preprint arXiv:2403.06892, 2024. 3

    2024 arXiv

  49. [57]

    Detrs beat yolos on real-time object detection

    Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. Detrs beat yolos on real-time object detection. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16965–16974, 2024. 3

    2024

  50. [58]

    Detecting twenty-thousand classes using image-level supervision

    Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr¨ahenb¨uhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In ECCV, pages 350–

  51. [59]

    Deformable detr: Deformable transformers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021. 4

    2021

  52. [60]

    Generalized decoding for pixel, image, and lan- guage

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and lan- guage. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages...

  53. [368]

    Springer, 2022. 2, 8

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.