Pith. sign in

REVIEW 4 major objections 5 minor 80 references

Improving Token-based Object Detection with Video

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Video objects can be encoded as discrete token sequences, letting a single autoregressive transformer do end-to-end video object detection and reach 91.14 mAP on UA-DETRAC.

desk verdict Tracklet tokenization with NA tokens is a genuinely new idea and the code is released, but the paper's own ablations undercut the headline claim that video helps, and the compute-bottleneck explanation for poor scaling is asserted, not tested. read the letter →

arxiv 2506.22562 v2 pith:YQFGZWF5 submitted 2025-06-27 cs.CV

Abhineet Singh , Nilanjan Ray This is my paper
classification cs.CV
keywords videoobjectdetectiontokenizationautoregressivetransformertrackletmulti-objecttrackingtemporalfusionUA-DETRACPix2Seq
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds on Pix2Seq, a language-modeling approach to object detection that represents each object as a short sequence of discrete tokens and generates them autoregressively. It extends that representation from single images to video by making each object one indivisible tracklet: for an N-frame window, an object is encoded as N four-token box tuples plus a class token, with a special NA token for frames where the object is absent. The authors claim this removes the need for anchor-box sampling and heuristics like non-maximum suppression, and that it lets the detector scale toward multi-object tracking simply by enlarging the temporal window. On UA-DETRAC, their late-fusion video model reaches 91.14 mAP, edging past the leading VSTAM model, but they report that the gain over the static baseline mostly comes from redundant overlapping windows and disappears when the stride equals the window length. They attribute the lack of further scaling with N to small batch sizes forced by GPU memory limits, not to the tokenization scheme.

What carries the argument

The load-bearing object is the tokenized tracklet: an N-frame object is written as a sequence of N four-tuples of quantized box corners plus one class token, giving 4N+1 tokens per object, with a reserved NA token standing for absence in any frame. This turns detection into next-token prediction, exactly like the static Pix2Seq baseline, so the same pretrained transformer encoder-decoder and vocabulary can be reused. The paper proposes three fusion points: early fusion inside a video Swin transformer backbone, middle fusion where per-frame self-attention is followed by pairwise compositional cross-attention between frames, and late fusion where per-frame features are concatenated with learned 3D positional embeddings so every output token can attend to every frame. The temporal stride T is the other mechanism: with T<N, overlapping windows give N-way redundant detections for interior frames, and merging them suppresses false negatives, which the paper identifies as the main source of the video model's margin over the static baseline.

What would settle it

Train the paper's video model and the static baseline on the same dataset with identical batch size, iterations, and GPU memory for N=4 and T=N; if longer windows still fail to beat the first-frame-only model or to improve over the static baseline, the bottleneck explanation is wrong.

Watch

Extended reading notes

Core claim

The paper's central claim is that video object detection can be done end-to-end by treating each moving object as a single, indivisible token sequence rather than as a set of frame-level boxes to be linked later. In this representation each object occupies 4N+1 tokens over an N-frame window: the discretized coordinates of its box in every frame, followed by its class token, with a dedicated NA token marking frames in which the object is absent, occluded, or not yet entered. The detector is therefore an autoregressive transformer that generates complete tracklets directly, so no proposal sampling, no background-box loss engineering, and no non-maximum suppression are required. On UA-DETRAC the best configuration (late fusion, N=2, trained on two A100 GPUs) achieves 91.14 mAP, exceeding VSTAM's 90.39 and the static baseline's 88.62. The paper also claims that the concept can generalize to multi-object tracking as N grows, while reporting that its own video models show no consistent improvement with N and that models fed only the first frame nearly match true video models; it reads both as symptoms of the compute and batch-size bottleneck rather than inherent limits of the approach.

Load-bearing premise

The paper's explanation that the video models' failure to improve with N is just a batch-size and capacity problem assumes the tracklet tokenization and fusion design are not themselves the limiting factor; this premise is not tested by any controlled comparison at equal batch size.

Editorial extensions

If this is right

  • If the central claim is right, video detection no longer needs anchor boxes, frame-by-frame linking, or non-maximum suppression: the same autoregressive decoder that produces boxes in Pix2Seq can emit entire tracklets as one sequence.
  • With sufficient GPU memory to increase N, the same framework should extend from windowed detection to multi-object tracking, since a tracklet spanning an entire video is already an identity-consistent trajectory.
  • Overlapping temporal windows (T<N) are a cheap accuracy lever: they give redundant detections per frame and improve performance without changing the model, which explains most of the reported gain over the static baseline.
  • Class-token weight equalization is a practical correction: it counteracts the 4N-fold over-weighting of localization tokens and improves classification, with the benefit growing with N.
  • The late-fusion architecture, which lets every output token attend to every frame, should be the preferred design once training can afford large batches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's first-frame-only experiments imply a testable corollary the authors leave implicit: on slowly moving scenes, a static detector trained to guess future-frame boxes may capture most of the benefit of video input, so the advantage of true video fusion should be sought in fast motion, long occlusions, and crowded scenes.
  • The tracklet-as-sequence view suggests a natural bridge from detection to tracking: if N covers a whole trajectory, object identity is encoded by sequence continuity, eliminating the separate association step; the paper stops short of claiming this.
  • The batch-size explanation is the paper's strongest falsifiable claim; a matched-batch-size comparison between the video model and the static baseline on the same hardware would decide whether the representation or the compute budget is the real ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper extends the Pix2Seq token-based object detector to video by representing each object tracklet as a sequence of 4N+1 discrete tokens over an N-frame temporal window, including a special NA token for missing frames. Three fusion architectures are proposed (early, middle, and late fusion), and experiments on ACAD, IPSC, and UA-DETRAC compare the resulting P2S-VID models against the static P2S baseline and several state-of-the-art video detectors. The headline result is a late-fusion N=2 model reaching 91.14 mAP on UA-DETRAC, slightly above the VSTAM leaderboard score, but the paper acknowledges that the improvement over P2S is largely due to redundant inference with T<N and that performance does not improve with N. The paper attributes the lack of scaling to limited batch size and computational resources, and it argues that with more resources the approach could scale to multi-object tracking.

Significance. The tokenization scheme is clean and the paper is unusually honest about its limitations; it ships code and models, and the ablations (class-token weight equalization, fusion architectures, static-input comparison) are informative. If the scalability claim were supported, the work would be a meaningful step toward unifying video detection and tracking within a language-modeling framework. However, the current evidence is conditional: the central scalability assertion rests on an untested compute-bottleneck assumption, and the empirical comparisons are weakened by test-set validation and a backbone confound on IPSC.

major comments (4)
  1. [IV-C2] The paper states that validation was performed directly on the test set, with model selection driven by test-set performance (and in some cases subsets of the test set). Because the paper compares many configurations (fusion scheme, N, stride, class equalization, 1D vs 2D tokens) and selects the 'best' for Tables 2-3 and Figs 8-15, the reported results are subject to selection bias. The plateau argument does not address architecture selection, and a held-out validation split or a pre-registered protocol is needed to support the claimed improvements.
  2. [IV-D3, Table 5] The IPSC comparison between P2S and P2S-VID is confounded by backbone choice: Table 5 shows that the static P2S baselines use a VIT-B backbone (with batch size 4) while the P2S-VID models use ResNet-50 (with batch sizes 12-16). The large gains reported for P2S-VID in Fig 9 could therefore reflect the backbone change and larger batch size rather than the video extension. A ResNet-50-based P2S baseline trained with the same frozen-backbone setting and comparable batch size is required to attribute the improvement to the video modeling.
  3. [V-C2, V-D, VI] The central scalability claim—that increasing N moves the method toward multi-object tracking—depends on the assertion that the flat or declining performance with N is entirely a compute/batch-size bottleneck. No controlled experiment at matched effective batch size (e.g., gradient accumulation or more GPUs for larger N) is reported. The static-input experiment in V-C2 shows that a model with access to only the first frame matches the video models even at N=8, which is equally consistent with the temporal fusion or tokenization design being the limiting factor. The paper's own caveat in V-C2 ('might not be good enough') leaves this alternative open, so the Sec VI conclusion that the bottleneck is 'very likely' a consequence of batch size is not established by the evidence. A batch-size-matched N-scaling study, or a comparison against a stronger temporal fusion baseline, is needed to support the scaling-to-MOT promise.
  4. [IV-D4, Tables 2-3] The headline UA-DETRAC result (91.14 mAP) is for a late-fusion N=2 model. Per the summary in Sec IV-D1, the improvement of P2S-VID over P2S is 'mainly due to the output redundancy ... obtained by using T < N'. Since leaderboard methods such as VSTAM do not use T<N redundant inference, the comparison in Table 2 conflates the model's temporal reasoning with an inference-time redundancy scheme. Reporting P2S-VID with T=N (no redundancy) would provide an apples-to-apples comparison and clarify whether the video model itself is competitive.
minor comments (5)
  1. [II-B1] In the occlusion example, 'tx 5' should be 'lx 5' to match the earlier notation.
  2. [IV-A, V-C2, III-B2] There are typos: 'folowing' should be 'following', 'adavantage' should be 'advantage', and 'unline' should be 'unlike'.
  3. [Table 1] The column headers (#1, #3, #4, Early-stage, Late-stage) are not clearly tied to the dataset names in the table body; please add clear grouping or a column with dataset names.
  4. [Fig 14] The caption says the left panel shows UA-DETRAC, but the displayed figure contains only IPSC panels; the figure and caption need to be reconciled.
  5. [II-B3] The 1D-coordinate variant uses both 'tl_i, br_i' and 'tli, bri' notation; please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the method is benchmarked externally and the main claims are stated as hypotheses, not derived from fitted inputs.

full rationale

The paper's derivation chain is a tokenization scheme (4N+1 tokens per tracklet), an architecture adaptation, and standard supervised training, followed by comparison against external methods. No parameter is fitted to a target quantity and then renamed as a prediction, and no equation is defined in terms of the result it is supposed to establish. The headline result of 91.14 mAP on UA-DETRAC is compared against the external VSTAM score of 90.39 on the public leaderboard, so the central empirical claim is externally falsifiable rather than forced by construction. The self-citations to the authors' earlier ACAD and IPSC papers provide datasets and metric definitions, but these are not load-bearing for the main claim: the primary comparisons are against the static Pix2Seq baseline and against independent state-of-the-art video detectors. The scalability-to-MOT statement is a conceptual extrapolation from the tracklet representation, not a fitted or data-derived prediction, and the paper does not claim to have demonstrated MOT. The oft-repeated batch-size bottleneck explanation for flat or declining performance with N is an untested assumption, and the paper even concedes that the fusion schemes 'might not be good enough' to use video information; that is a correctness or evidence weakness, not circularity. The paper also explicitly concedes that the observed improvement over the baseline is 'mainly due to the output redundancy ... obtained by using T < N' and that the advantage mostly disappears at T = N, which further separates the representational claim from the benchmark gain. Validation was performed on test sets rather than a held-out validation split, which is a methodological concern, but it does not make any result equivalent to its input by construction. Overall, the claim chain is self-contained and externally benchmarked, so no circular step is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim does not rest on a fitted physical constant; the free parameters are architecture and training choices. The NA token is a design artifact, not an independently evidenced entity. The main load-bearing assumptions are the transferability of pretrained static weights and the sufficiency of the proposed fusion schemes, both empirically motivated rather than proven.

free parameters (5)
  • N (temporal window length) = 2 to 32; best reported results at N=2
    Chosen by hand; main results use N=2; performance does not consistently improve with N (Sec V-D).
  • T (temporal stride) = 1 for training; 1 and N for inference
    Chosen by hand; overlapping windows (T=1) drive most of the reported improvement over static detection (Sec IV-D1).
  • G (frame gap) = 1
    Chosen by hand; larger values proposed but not tested (Sec II-B4).
  • H (coordinate bins) = 2000 (2K)
    Chosen by hand; limits localization granularity; the 1D variant drops to H=160 with worse results (Sec V-E).
  • Class token weight equalization = enabled or disabled per model
    A design choice that materially changes classification performance (Sec V-A); not a scalar but a training objective modifier.
assumptions (4)
  • standard math Transformer self-attention and autoregressive token prediction are adequate for detection.
    Adopted from Pix2Seq (Ref [1]) and Vaswani et al. (Ref [4]).
  • domain assumption Discretizing bounding box coordinates into H bins preserves localization accuracy needed for the target datasets.
    Sec II-A; the paper relies on this tokenization throughout.
  • ad hoc to paper Pairwise compositional cross-MHA with shared weights is a sufficient temporal fusion mechanism.
    Chosen because the hierarchical variant was unstable (Sec III-B2), not from a theoretical guarantee.
  • ad hoc to paper Freezing the pretrained ResNet-50 backbone preserves enough feature quality for video detection.
    Empirically found to work for middle/late fusion (Sec V-C1), but not justified beyond these experiments.
invented entities (1)
  • NA token
    purpose: Denotes object absence in a frame (entry, exit, or occlusion) within the temporal window.
    A vocabulary token introduced in Sec II-B1; its utility is only demonstrated within this paper's models and it has no independently falsifiable handle outside the proposed method.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Token-based Object Detection with Video." pith.science (2026). https://pith.science/paper/YQFGZWF5

@misc{pith2026250622562,
  author       = {Pith},
  title        = {Pith review of: Improving Token-based Object Detection with Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YQFGZWF5}},
  note         = {Machine review of arXiv:2506.22562}
}
read the original abstract

This paper improves upon the Pix2Seq object detector by extending it for videos. In the process, it introduces a new way to perform end-to-end video object detection that improves upon existing video detectors in two key ways. First, by representing objects as variable-length sequences of discrete tokens, we can succinctly represent widely varying numbers of video objects, with diverse shapes and locations, without having to inject any localization cues in the training process. This eliminates the need to sample the space of all possible boxes that constrains conventional detectors and thus solves the dual problems of loss sparsity during training and heuristics-based postprocessing during inference. Second, it conceptualizes and outputs the video objects as fully integrated and indivisible 3D boxes or tracklets instead of generating image-specific 2D boxes and linking these boxes together to construct the video object, as done in most conventional detectors. This allows it to scale effortlessly with available computational resources by simply increasing the length of the video subsequence that the network takes as input, even generalizing to multi-object tracking if the subsequence can span the entire video. We compare our video detector with the baseline Pix2Seq static detector on several datasets and demonstrate consistent improvement, although with strong signs of being bottlenecked by our limited computational resources. We also compare it with several video detectors on UA-DETRAC to show that it is competitive with the current state of the art even with the computational bottleneck. We make our code and models publicly available.

Figures

Figures reproduced from arXiv: 2506.22562 by the authors.

Figure 1
Figure 1. FIGURE 1: Visualization of Pix2Seq object tokenization on images from (left) UA-DETRAC [ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. FIGURE 2: Visualization of video object tokenization on video clips from (left) UA-DETRAC and (right) IPSC datasets, both [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. FIGURE 3: Flow diagram representing the high-level processing in the Pix2Seq encoder on the left and decoder on the right. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: FIGURE 5: Flow diagram representing high-level processing [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: FIGURE 6: Flow diagram for the pairwise compositional [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: FIGURE 7: Flow diagram representing high-level processing in the late-fusion video encoder and decoder on the left and right [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: FIGURE 8: Results on ACAD configurations [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: FIGURE 9: Comparing P2S and P2S-VID on IPSC dataset in terms of (left) high-level and (right) low-level detection metrics. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: FIGURE 10: Performance impact of equalizing the class token weights on (left) ACAD [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: FIGURE 11: Impact of batch size on validation performance when training (top) P2S and (bottom) P2S-VID on ACAD #3. The [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: FIGURE 12: Comparing the three video architectures (Sec. [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: FIGURE 13: Performance impact of replacing [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: summarizes the impact of video length N and stride T on video detection performance over both UA-DETRAC and IPSC late-stage datasets. Neither dataset shows any consistent improvement in performance with N. Quite the contrary, in fact. The only case that shows any sign…
Figure 15
Figure 15. Figure 15: FIGURE 15: Comparing the standard 2D coordinate tokenization of P2S-VID with its 1D variant on the (left) early and (right) [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: FIGURE 16: Flow diagram for the hierarchical variant of the [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

80 extracted references · 73 canonical work pages

  1. [1]

    T. Chen, S. Saxena, L. Li, D. J. Fleet, and G. E. Hinton, ‘‘Pix2seq: A Language Modeling Framework for Object Detection,’’ in Proceedings of the 10th International Conference on Learning Representations (ICLR) , 2022

    work page 2022

  2. [2]

    Singh, ‘‘P2S-Video: Extension of Pix2Seq for Video Detection and Segmentation,’’ online: https://github.com/abhineet123/p2s-video

    A. Singh, ‘‘P2S-Video: Extension of Pix2Seq for Video Detection and Segmentation,’’ online: https://github.com/abhineet123/p2s-video

    work page

  3. [3]

    Krizhevsky, I

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, ‘‘ImageNet Classification with Deep Convolutional Neural Networks,’’ Advances in neural informa- tion processing systems, vol. 25, 2012

    work page 2012

  4. [4]

    V aswani, N

    A. V aswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ‘‘Attention is All you Need,’’ NIPS, 2017

    work page 2017

  5. [5]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Un- terthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, ‘‘An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,’’ ICLR, 2021

    work page 2021

  6. [6]

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, ‘‘Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,’’ ICCV, pp. 9992–10 002, 2021

    work page 2021

  7. [7]

    Girshick, ‘‘Fast R-CNN,’’ in ICCV, Dec 2015, pp

    R. Girshick, ‘‘Fast R-CNN,’’ in ICCV, Dec 2015, pp. 1440–1448

    work page 2015

  8. [8]

    S. Ren, K. He, R. Girshick, and J. Sun, ‘‘Faster R-CNN: Towards Real- Time Object Detection with Region Proposal Networks,’’ IEEE Transac- tions on Pattern Analysis and Machine Intelligence , vol. 39, no. 6, pp. 1137–1149, June 2017

    work page 2017

Show all 80 references
  1. [9]

    W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed, C.-Y . Fu, and A. C. Berg, ‘‘SSD: Single Shot MultiBox Detector,’’ in ECCV, 2016

    2016

  2. [10]

    T. Lin, P . Goyal, R. Girshick, K. He, and P . Dollar, ‘‘Focal Loss for Dense Object Detection,’’ in ICCV, Oct 2017, pp. 2999–3007

    2017

  3. [11]

    K. He, G. Gkioxari, P . Dollár, and R. B. Girshick, ‘‘Mask R-CNN,’’TPAMI, vol. 42, pp. 386–397, 2020

    2020

  4. [12]

    Cai and N

    Z. Cai and N. V asconcelos, ‘‘Cascade R-CNN: High Quality Object Detec- tion and Instance Segmentation,’’ TPAMI, vol. 43, pp. 1483–1498, 2021

    2021

  5. [13]

    Redmon, S

    J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi, ‘‘Y ou Only Look Once: Unified, Real-Time Object Detection,’’ CVPR, pp. 779–788, 2015

    2015

  6. [14]

    Redmon and A

    J. Redmon and A. Farhadi, ‘‘YOLO9000: Better, Faster, Stronger,’’ CVPR, pp. 6517–6525, 2017

    2017

  7. [15]

    abs/1804.02767, 2018

    ——, ‘‘YOLOv3: An Incremental Improvement,’’ CoRR, vol. abs/1804.02767, 2018. [Online]. Available: http://arxiv.org/abs/1804. 02767

    2018 arXiv

  8. [16]

    Bochkovskiy, C.-Y

    A. Bochkovskiy, C.-Y . Wang, and H.-Y . M. Liao, ‘‘YOLOv4: Optimal speed and accuracy of object detection,’’ArXiv, vol. abs/2004.10934, 2020

    2004 arXiv

  9. [17]

    G. R. Jocher, A. Stoken, J. Borovec, NanoCode, A. Chaurasia, TaoXie, C. Liu, Abhiram, Laughing, tkianai, yxNONG, A. Hogan, lorenzomammana, AlexWang, J. Hájek, L. Diaconu, Marc, Y . Kwon, Oleg, wanghaoyang, Y . Defretin, A. Lohia, ml ah, B. Milanko, B. Fineran, D. P . Khromov, ...

    2021

  10. [18]

    C.-Y . Wang, A. Bochkovskiy, and H.-Y . M. Liao, ‘‘YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors,’’ 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 7464–7475, 2022. [Online]. Available: https://api.s...

    2023

  11. [19]

    G. R. Jocher, A. Stoken, J. Borovec, NanoCode, A. Chaurasia, TaoXie, C. Liu, Abhiram, Laughing, tkianai, yxNONG, A. Hogan, lorenzomammana, AlexWang, J. Hájek, L. Diaconu, Marc, Y . Kwon, Oleg, wanghaoyang, Y . Defretin, A. Lohia, ml ah, B. Milanko, B. Fineran, D. P . Khromov, ...

    2023

  12. [20]

    Wang, I.-H

    C.-Y . Wang, I.-H. Y eh, and H. Liao, ‘‘YOLOv9: Learning What Y ou Want to Learn Using Programmable Gradient Information,’’ ArXiv, vol. abs/2402.13616, 2024. [Online]. Available: https://api.semanticscholar. org/CorpusID:267770251

    2024 arXiv

  13. [21]

    J. R. Terven, D.-M. Córdova-Esparza, and J.-A. Romero-González, ‘‘A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS,’’ Mach. Learn. Knowl. Extr . , vol. 5, pp. 1680–1716, 2023

    2023

  14. [22]

    Shrivastava, A

    A. Shrivastava, A. K. Gupta, and R. B. Girshick, ‘‘Training Region-Based Object Detectors with Online Hard Example Mining,’’CVPR, pp. 761–769, 2016

    2016

  15. [23]

    Bewley, Z

    A. Bewley, Z. Ge, L. Ott, F. T. Ramos, and B. Upcroft, ‘‘Simple Online and Realtime Tracking,’’ ICIP, pp. 3464–3468, 2016

    2016

  16. [24]

    Bochinski, V

    E. Bochinski, V . Eiselein, and T. Sikora, ‘‘High-Speed Tracking-by- Detection without using Image Information,’’ A VSS, pp. 1–6, 2017

    2017

  17. [25]

    Chen, ‘‘Pix2Seq Codebase: Multi-tasks with generative modeling (au- toregressive and diffusion),’’ online: https://github.com/google-research/ pix2seq

    T. Chen, ‘‘Pix2Seq Codebase: Multi-tasks with generative modeling (au- toregressive and diffusion),’’ online: https://github.com/google-research/ pix2seq

  18. [26]

    T. Chen, S. Saxena, L. Li, T.-Y . Lin, D. J. Fleet, and G. Hinton, ‘‘A Unified Sequence Interface for Vision Tasks,’’ in Proceedings of the 36th Annual Conference on Neural Information Processing Systems (NIPS) , 2022

    2022

  19. [27]

    T. Chen, R. Zhang, and G. Hinton, ‘‘Analog bits: Generating discrete data using diffusion models with self-conditioning,’’ in Proceedings of the 11th International Conference on Learning Representations (ICLR) , 2023

    2023

  20. [28]

    T. Chen, L. Li, S. Saxena, G. E. Hinton, and D. J. Fleet, ‘‘A General- ist Framework for Panoptic Segmentation of Images and Videos,’’ 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 909– 919, 2023

    2023

  21. [29]

    X. Chen, H. Peng, D. Wang, H. Lu, and H. Hu, ‘‘SeqTrack: Sequence to Se- quence Learning for Visual Object Tracking,’’2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 14 572–14 581, 2023

    2023

  22. [30]

    Zheng, B

    Y . Zheng, B. Zhong, Q. Liang, G. Li, R. Ji, and X. Li, ‘‘Toward Unified Token Learning for Vision-Language Tracking,’’ IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, pp. 2125–2135, 2023

    2023

  23. [31]

    J. Liu, H. Ding, Z. Cai, Y . Zhang, R. K. Satzoda, V . Mahadevan, and R. Manmatha, ‘‘Polyformer: Referring image segmentation as sequential 16 VOLUME 11, 2023 Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS TABLE 5: Details of the models whose results a...

    2023

  24. [32]

    C. Zhu, Y . Zhou, Y . Shen, G. Luo, X. Pan, M. Lin, C. Chen, L. Cao, X. Sun, and R. Ji, ‘‘SeqTR: A simple yet universal network for visual grounding,’’ in European Conference on Computer Vision . Springer, 2022, pp. 598– 615

    2022

  25. [33]

    Y ang, Z

    Z. Y ang, Z. Gan, J. Wang, X. Hu, F. Ahmed, Z. Liu, Y . Lu, and L. Wang, ‘‘Unitab: Unifying text and box outputs for grounded vision-language modeling,’’ inEuropean Conference on Computer Vision. Springer, 2022, pp. 521–539

    2022

  26. [34]

    S. H. Rezatofighi, V . K. Bg, A. Milan, E. Abbasnejad, A. Dick, and I. Reid, ‘‘Deepsetnet: Predicting sets with deep neural networks,’’ in 2017 IEEE International Conference on Computer Vision (ICCV) . IEEE, 2017, pp. 5257–5266

    2017

  27. [35]

    Pineda, A

    L. Pineda, A. Salvador, M. Drozdzal, and A. Romero, ‘‘Elucidating Image- to-Set Prediction: An Analysis of Models, Losses and Datasets,’’ arXiv preprint arXiv:1904.05709, 2019

    1904 arXiv

  28. [36]

    Y . Xue, J. Mao, M. Niu, H. Xu, M. B. Mi, W. Zhang, X. Wang, and X. Wang, ‘‘Point2Seq: Detecting 3D Objects as Sequences,’’ 2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR) , pp. 8511– 8520, 2022

    2022

  29. [37]

    Z. Chen, Y . Zhu, Z. Li, F. Y ang, W. Li, H. Wang, C. Zhao, L. Wu, R. Zhao, J. Wang et al., ‘‘Obj2seq: Formatting objects as sequences with class prompt for visual tasks,’’Advances in Neural Information Processing Systems, vol. 35, pp. 2494–2506, 2022

    2022

  30. [38]

    Carion, F

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, ‘‘End-to-End Object Detection with Transformers,’’ ECCV, 2020

    2020

  31. [39]

    X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, ‘‘Deformable DETR: Deformable Transformers for End-to-End Object Detection,’’ ICLR, 2021

    2021

  32. [40]

    Kolesnikov, A

    A. Kolesnikov, A. Susano Pinto, L. Beyer, X. Zhai, J. Harmsen, and N. Houlsby, ‘‘UViM: A unified modeling approach for vision with learned guiding codes,’’ Advances in Neural Information Processing Systems , vol. 35, pp. 26 295–26 308, 2022

    2022

  33. [41]

    J. Ning, C. Li, Z. Zhang, C. Wang, Z. Geng, Q. Dai, K. He, and H. Hu, ‘‘All in tokens: Unifying output space of visual tasks via soft token,’’ in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 19 900–19 910

    2023

  34. [42]

    T , ucudean, M

    G. T , ucudean, M. Bucos, B. Drăgulescu, and C. D. Căleanu, ‘‘Natural Language Processing with Transformers: A Review,’’ PeerJ Computer Science, vol. 10, 2024

    2024

  35. [43]

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong, Y . Du, C. Y ang, Y . Chen, Z. Chen, J. Jiang, R. Ren, Y . Li, X. Tang, Z. Liu, P . Liu, J. Nie, and J. rong Wen, ‘‘A Survey of Large Language Models,’’ ArXiv, vol. abs/2303.18223, 2023

    2023 arXiv

  36. [44]

    Minaee, T

    S. Minaee, T. Mikolov, N. Nikzad, M. A. Chenaghlu, R. Socher, X. Am- atriain, and J. Gao, ‘‘Large Language Models: A Survey,’’ ArXiv, vol. abs/2402.06196, 2024

    2024 arXiv

  37. [45]

    Radford and K

    A. Radford and K. Narasimhan, ‘‘Improving Language Understanding by Generative Pre-Training,’’ 2018

    2018

  38. [46]

    Fujitake and A

    M. Fujitake and A. Sugimoto, ‘‘Video sparse transformer with attention- guided memory for video object detection,’’ IEEE Access , vol. 10, pp. 65 886–65 900, 2022

    2022

  39. [47]

    H. Wang, J. Tang, X. Liu, S. Guan, R. Xie, and L. Song, ‘‘PTSEFormer: Progressive Temporal-Spatial Enhanced TransFormer Towards Video Ob- ject Detection,’’ in European Conference on Computer Vision. Springer, 2022, pp. 732–747

    2022

  40. [48]

    Q. Zhou, X. Li, L. He, Y . Y ang, G. Cheng, Y . Tong, L. Ma, and D. Tao, ‘‘TransVOD: End-to-end video object detection with spatial-temporal transformers,’’ IEEE Transactions on Pattern Analysis and Machine In- telligence, vol. 45, no. 6, pp. 7853–7869, 2022

    2022

  41. [49]

    C. Deng, D. Chen, and Q. Wu, ‘‘Identity-Consistent Aggregation for Video Object Detection,’’ 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 13 388–13 398, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:260900188

    2023

  42. [50]

    Qi and X

    Q. Qi and X. Wang, ‘‘Tgbformer: Transformer-graphformer blender network for video object detection,’’ in AAAI Conference on Artificial Intelligence, 2025. [Online]. Available: https://api.semanticscholar.org/ CorpusID:277103597

    2025

  43. [51]

    Y . Chen, Y . Cao, H. Hu, and L. Wang, ‘‘Memory enhanced global-local aggregation for video object detection,’’ in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 10 337– 10 346

    2020

  44. [52]

    C. Xu, J. Zhang, M. Wang, G. Tian, and Y . Liu, ‘‘Multilevel spatial- temporal feature aggregation for video object detection,’’ IEEE Transac- tions on Circuits and Systems for Video Technology , vol. 32, no. 11, pp. VOLUME 11, 2023 17 Author et al.: Preparation of Papers for ...

    2023

  45. [53]

    Han and Z

    L. Han and Z. Yin, ‘‘Global memory and local continuity for video object detection,’’ IEEE Transactions on Multimedia , vol. 25, pp. 3681–3693, 2022

    2022

  46. [54]

    Roh and K.-S

    S.-D. Roh and K.-S. Chung, ‘‘DAFA: Diversity-Aware Feature Aggregation for Attention-Based Video Object Detection,’’ IEEE Access, vol. 10, pp. 93 453–93 463, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:252022543

    2022

  47. [55]

    L. Han, P . Wang, Z. Yin, F. Wang, and H. Li, ‘‘Class-Aware Feature Aggregation Network for Video Object Detection,’’ IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, pp. 8165–8178, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:240730954

    2022

  48. [56]

    K. A. Hashmi, D. Stricker, and M. Z. Afzal, ‘‘Spatio-Temporal Learnable Proposals for End-to-End Video Object Detection,’’ in British Machine Vision Conference, 2022. [Online]. Available: https://api.semanticscholar. org/CorpusID:252715466

    2022

  49. [57]

    F. He, N. Gao, J. Jia, X. Zhao, and K. Huang, ‘‘QueryProp: Object Query Propagation for High-Performance Video Object Detection,’’ in Proceed- ings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 1, 2022, pp. 834–842

    2022

  50. [58]

    P . Sun, R. Zhang, Y . Jiang, T. Kong, C. Xu, W. Zhan, M. Tomizuka, L. Li, Z. Y uan, C. Wang et al. , ‘‘Sparse R-CNN: End-to-end object detection with learnable proposals,’’ in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 14 454–14 463

    2021

  51. [59]

    Sabater, L

    A. Sabater, L. Montesano, and A. C. Murillo, ‘‘Robust and efficient post- processing for video object detection,’’ in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2020, pp. 10 536–10 542

    2020

  52. [60]

    Y . Shi, N. Wang, and X. Guo, ‘‘YOLOV: Making still image object detec- tors great at video object detection,’’ inProceedings of the AAAI conference on artificial intelligence, vol. 37, no. 2, 2023, pp. 2254–2262

    2023

  53. [61]

    Y . Shi, T. Zhang, and X. Guo, ‘‘Practical Video Object Detection via Feature Selection and Aggregation,’’ arXiv preprint arXiv:2407.19650 , 2024

    2024 arXiv

  54. [62]

    Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, ‘‘Y olox: Exceeding yolo series in 2021,’’ arXiv preprint arXiv:2107.08430, 2021

    2021 arXiv

  55. [63]

    Ehteshami Bejnordi, A

    B. Ehteshami Bejnordi, A. Habibian, F. Porikli, and A. Ghodrati, ‘‘SAL- ISA: Saliency-based input sampling for efficient video object detection,’’ in European Conference on Computer Vision. Springer, 2022, pp. 300–316

    2022

  56. [64]

    M. Tan, R. Pang, and Q. V . Le, ‘‘EfficientDet: Scalable and efficient object detection,’’ inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10 781–10 790

    2020

  57. [65]

    Singh, ‘‘Video Detection and Segmentation with Language Modeling,’’ online: https://webdocs.cs.ualberta.ca/~asingh1/p2s/

    A. Singh, ‘‘Video Detection and Segmentation with Language Modeling,’’ online: https://webdocs.cs.ualberta.ca/~asingh1/p2s/

  58. [66]

    L. Wen, D. Du, Z. Cai, Z. Lei, M.-C. Chang, H. Qi, J. Lim, M.-H. Y ang, and S. Lyu, ‘‘UA-DETRAC: A new benchmark and protocol for multi-object detection and tracking,’’ CVIU, vol. 193, p. 102907, 2020

    2020

  59. [67]

    Singh, I

    A. Singh, I. Jasra, O. Mouhammed, N. Dadheech, N. Ray, and J. Shapiro, ‘‘Towards Early Prediction of Human iPSC Reprogramming Success,’’ Machine Learning for Biomedical Imaging , vol. 2, pp. 390–407, 2023. [Online]. Available: https://melba-journal.org/2023:014

    2023

  60. [68]

    K. Kang, W. Ouyang, H. Li, and X. Wang, ‘‘Object Detection from Video Tubelets with Convolutional Neural Networks,’’CVPR, pp. 817–825, 2016

    2016

  61. [69]

    K. He, X. Zhang, S. Ren, and J. Sun, ‘‘Deep Residual Learning for Image Recognition,’’ CVPR, pp. 770–778, 2016

    2016

  62. [70]

    Z. Liu, J. Ning, Y . Cao, Y . Wei, Z. Zhang, S. Lin, and H. Hu, ‘‘Video Swin Transformer,’’2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 3192–3201, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:235624247

    2022

  63. [71]

    Feichtenhofer, H

    C. Feichtenhofer, H. Fan, J. Malik, and K. He, ‘‘SlowFast Networks for Video Recognition,’’ 2019 IEEE/CVF International Conference on Com- puter Vision (ICCV), pp. 6201–6210, 2018

    2019

  64. [72]

    Innat, ‘‘Keras 3 Implementation of Video Swin Transformers for 3D Video Modeling,’’ online: https://github.com/innat/VideoSwin, December 2024

    M. Innat, ‘‘Keras 3 Implementation of Video Swin Transformers for 3D Video Modeling,’’ online: https://github.com/innat/VideoSwin, December 2024

    2024

  65. [73]

    W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijaya- narasimhan, F. Viola, T. Green, T. Back, A. Natsev, M. Suleyman, and A. Zisserman, ‘‘The Kinetics Human Action Video Dataset,’’ ArXiv, vol. abs/1705.06950, 2017

    2017 arXiv

  66. [74]

    Singh, M

    A. Singh, M. Pietrasik, G. Natha, N. Ghouaiel, K. Brizel, and N. Ray, ‘‘Animal Detection in Man-made Environments,’’ 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) , pp. 1427–1438, 2020. [Online]. Available: https://api.semanticscholar.org/ CorpusID:204900826

    2020

  67. [75]

    ‘‘Object Detection on UA-DETRAC,’’ online: https://paperswithcode.com/ sota/object-detection-on-ua-detrac

  68. [76]

    Fujitake and A

    M. Fujitake and A. Sugimoto, ‘‘Temporal feature enhancement network with external memory for object detection in surveillance video,’’ in 2020 25th International Conference on Pattern Recognition (ICPR) . IEEE, 2021, pp. 7684–7691

    2020

  69. [77]

    Kim, P .-K

    K.-J. Kim, P .-K. Kim, Y .-S. Chung, and D.-H. Choi, ‘‘Performance en- hancement of YOLOv3 by adding prediction layers with spatial pyramid pooling for vehicle detection,’’ in2018 15th IEEE international conference on advanced video and signal based surveillance (A VSS) . IEEE...

    2018

  70. [78]

    ——, ‘‘Multi-scale detector for accurate vehicle detection in traffic surveil- lance data,’’ IEEE Access, vol. 7, pp. 78 311–78 319, 2019

    2019

  71. [79]

    Perreault, G.-A

    H. Perreault, G.-A. Bilodeau, N. Saunier, and M. Héritier, ‘‘SpotNet: Self- attention multi-task network for object detection,’’ in2020 17th Conference on Computer and Robot Vision (CRV) . IEEE, 2020, pp. 230–237

    2020

  72. [80]

    ——, ‘‘FFA VOD: Feature fusion architecture for video object detection,’’ Pattern Recognition Letters, vol. 151, pp. 294–301, 2021. ABHINEET SINGH received the B.Tech. degree in information technology from IIIT Allahabad, Prayagraj, India, in 2013, the M.Sc. degree in computing...

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.