Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Semantic multi-object tracking should be generative: a vision-language model can narrate every tracked object, and social interactions can be deduced from dense text rather than trained as labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 11:21 UTC pith:OHGDPP3B

load-bearing objection A substantial benchmark and a genuinely new MLLM-driven SMOT framework, but the headline claims of SOTA tracking and a qualitative semantic leap rest on an apples-to-oranges tracking comparison and a decoupled evaluation protocol that masks tracking failures. the 4 major comments →

arxiv 2601.06550 v4 pith:OHGDPP3B submitted 2026-01-10 cs.CV cs.AI

Generative Semantic Multi-Object Tracking: A Large-Scale Benchmark and an MLLM-Driven Reasoning Framework

Pan Liao , Feng Yang , Di Wu , Jinwen Yu , Wang Zhao , Dingwen Zhang This is my paper
classification cs.CV cs.AI
keywords semantic multi-object trackingmulti-modal large language modeldense video captioningopen-world trackingvideo understandingtemporal hallucinationemergent interaction reasoningGrand-SMOT benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Semantic multi-object tracking today answers 'where are the objects' and, at best, attaches a small set of predefined interaction tags. This paper sets out to move the task past that ceiling by making it a generative reasoning problem. It introduces Grand-SMOT, a large open-world benchmark whose videos come with two streams of dense captions — a video-level description of the environment and per-instance descriptions of each target's behavior — and LLMTrack, a framework that feeds a vision-language model (a multi-modal large language model) compressed trajectory tokens so it can produce a running, context-aware narrative for every tracked object while still associating identities over time. The paper's central claim is that with such dual-stream supervision, a language model can deduce social interactions from text alone, making interaction labels and dedicated interaction-recognition branches unnecessary. If that is right, multi-object tracking stops being a geometric preprocessing step and becomes a source of open-ended video understanding, which matters for embodied agents, long-video reasoning, and downstream world-model building.

Core claim

On its own terms, the paper's discovery is that semantic multi-object tracking can be reframed as open-ended generative reasoning, and that the right data and a simple architectural ordering unlock it. LLMTrack compresses discrete geometric trajectories into continuous tokens via a Spatio-Temporal Fusion Module — a Video Fusion branch maintains a recurrent memory of global scene context, and an Instance Fusion branch aggregates each target's recent embeddings with adaptive attention. By placing the global context token before the instance tokens in the prompt, the 'Macro-Understanding-First' mechanism forces every per-object description to be conditioned on the environment. Trained in three

What carries the argument

Central mechanism: the Spatio-Temporal Fusion Module, which compresses discrete trajectories into continuous semantic tokens. Video Fusion keeps a recurrent memory token of the global scene via cross-attention; Instance Fusion aggregates each target's sliding-window embeddings via adaptive additive attention. Fed to the LLM with the global token first (Macro-Understanding-First), these tokens let the model generate per-object narratives conditioned on context, suppressing temporal hallucination. The companion machinery is Grand-SMOT's dual-stream annotation scheme (video-level context + instance-level behavior) and the three-stage training recipe that aligns geometric features with language

Load-bearing premise

The load-bearing premise is that the decoupled evaluation protocol in Appendix G.2 — scoring only detected objects and taking the best caption over ID-switched fragments, with model-generated captions as ground truth — is a valid measure of semantic understanding; if that premise fails, the reported qualitative leap is partly an artifact.

What would settle it

Re-score LLMTrack and baselines on the same benchmark without the two masking operations: count every undetected target as a zero caption score and assign the lowest (not the highest) score across an ID-switched tracklet's fragments. If the margin over SMOTer and other baselines collapses or reverses, the semantic 'qualitative leap' is an artifact of the decoupled protocol rather than a true generative understanding gain.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • SMOT no longer needs hand-defined interaction taxonomies: a language model can deduce relations from dense descriptions of individuals and their environment.
  • A single architecture can carry geometric association and open-ended narrative generation, letting downstream systems consume tracked objects as described entities rather than bare boxes.
  • The dual-stream caption format (micro-behavior plus macro-context) becomes a reusable supervision template for training long-video reasoning models.
  • Investing in caption density and veracity can substitute for building interaction-specific recognition heads, pointing data effort toward narrative quality.
  • Scaling the language model from 0.5B to 4B improves semantic understanding without hurting tracking, suggesting a practical scaling path for cognitive trackers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: Because the semantic metrics in Appendix G.2 are computed only over detected objects and take the maximum caption score over ID-switched fragments, an end-to-end evaluation that penalizes every missed object and identity break would likely compress the reported gap over baselines; the 'qualitative leap' may be partly an artifact of decoupling.
  • Inference: The paper's interaction-deduction experiment suggests a testable scaling hypothesis: if caption density and fidelity are increased while the architecture is held fixed, zero-shot deduction of relations should improve monotonically — a prediction the benchmark could be used to verify.
  • Inference: The Macro-Understanding-First ordering is a simple inductive bias that could transfer to other streaming video-language tasks (dense video captioning, embodied instruction following), where global context should anchor local predictions.
  • Inference: Since the ground truth was generated by an LLM with manual correction on only 255 of 4,770 sequences, a fully human-annotated test subset would provide a stronger target for assessing the 'qualitative leap' claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Grand-SMOT, a large-scale semantic multi-object tracking benchmark built by re-annotating BenSMOT and TAO with dual-stream dense narratives (video-level environment and instance-level behavior), and proposes LLMTrack, an MLLM-based framework that couples a Grounding DINO detector/ByteTrack-style association with a spatio-temporal fusion module and an LLaVA-OneVision language backend. The authors report state-of-the-art geometric tracking on BenSMOT (75.23 HOTA) and large gains over prior work on semantic captioning metrics, and they claim that interaction understanding can emerge from text-only deduction given good macro and micro descriptions. The appendix describes a decoupled semantic evaluation protocol, dataset construction details, and an 'emergent deduction' experiment.

Significance. If the claims held, this would be a useful step toward open-ended generative semantic tracking: Grand-SMOT is larger and semantically denser than BenSMOT, and the three-stage training recipe (tracking warm-up, fusion alignment, LoRA LLM tuning with decoupled TBPTT) is a plausible engineering contribution. The paper also makes an interesting conceptual argument that interaction labels should be deducible from behavior-plus-context narratives. However, the main evidence is weakened by three load-bearing problems: the semantic evaluation masks tracking errors, the headline geometric comparison mixes training data and detector choices, and the 'emergent deduction' experiment risks circularity because the narratives were generated conditionally on the very interaction tags being deduced. The benchmark and framework may be salvageable, but the current manuscript does not substantiate the abstract's 'qualitative leap' claim.

major comments (4)
  1. [Appendix G.2 and Table 3] The semantic scores central to the paper are computed under a decoupled protocol that (i) averages only over detected GT objects (Final_Caption_Score = 1/|D| sum_{i in D} Metric(C_i, C^GT_i)) and (ii) for each GT trajectory fragmented by ID switches takes the maximum score over matched predicted tracklets (M(C_GT) = max_i Metric(C_i, C_GT)). This removes the exact tracking failures that the authors themselves identify in G.1 as disrupting the continuous semantic narrative. Consequently Table 3 does not measure end-to-end 'dynamic semantic reasoning' as claimed in the Abstract/Conclusion; it measures best-fragment conditional captioning on the subset of objects that were detected. A model that narrates easy objects well while missing hard objects, or that describes a target correctly in one fragment before losing it, receives full semantic credit. The paper should either report coupled me
  2. [§5.2 Table 2 vs §5.3 Table 4 and Appendix D.1 Table 11] The headline 75.23 HOTA is obtained with the model trained on BenSMOT + TAO (Table 4, '+TAO' row), while the same architecture trained on BenSMOT alone yields 74.61 HOTA (Tables 4 and 11). The baselines in Table 2 are not given TAO training, and the caption states they use their officially optimized detectors while LLMTrack uses Grounding DINO. The reported geometric advantage therefore conflates extra training data and a stronger open-vocabulary detector with the tracking architecture itself. A matched-data, matched-detector comparison is needed before claiming state-of-the-art geometric tracking. The BenSMOT-only result in Table 11 is a step in the right direction, but the main table and abstract use the +TAO number without the corresponding caveat.
  3. [§3.2 vs Appendix A.2, Table 7] The 'emergent deduction' experiment is circular in its current form. BenSMOT's sparse interaction tags are used as semantic anchors to generate the dense narratives in §3.2; the 'GT Text' used in the text-only deduction experiment is exactly this generated text. Recovering an interaction category from text that was generated conditionally on that category does not demonstrate emergent visual reasoning, nor does it prove that interaction is deducible from independently obtained behavior and environment descriptions. The comparison against explicit feature fusion in Table 7 should be redone with human-verified captions, or with captions generated while withholding the interaction label, and the prediction target should be held out from the generation prompt.
  4. [§3.3, Appendix E.3, and Table 3] Only 255 of 4,770 sequences received manual correction; the remaining semantic references are Qwen3-VL-32B outputs. The semantic metrics in Table 3 are then computed by an LLM judge (GPT-4o, or a committee in Appendix D.4) comparing model outputs against these LLM-generated references. This makes the absolute GPT-S and n-gram numbers partly a measure of agreement with a specific MLLM's writing style, and it weakens the benchmark's claim to provide human-level physical correctness. The authors should report results on the manually corrected subset separately, and provide human-correlation calibration for each data split rather than only a global Spearman correlation.
minor comments (4)
  1. [§5.2 Table 2 / Appendix D.1] SMOTer is reported as 71.55 HOTA in Table 2 and 71.98 in Table 11, and LLMTrack is 75.23 in Table 2 but 74.61 in Table 11. Please unify numbers and clearly state which training/evaluation setting each table refers to.
  2. [§5.2 Table 2] The abstract and conclusion claim 'state-of-the-art geometric tracking robustness' without qualification, but Table 2 shows OVTrack* achieving higher TETA (34.7 vs 33.6) on the TAO split. The SOTA claim should be restricted to the BenSMOT split or otherwise justified.
  3. [Figure 2 and Appendix H] Several prompts and captions contain typos, e.g., 'Promat for Expansion', 'Promat for Generation', 'Promat for Fusion'. A careful proofreading pass is needed.
  4. [Eq. (8), §4.2] The video loss lambda_vid appears only at the final clip k=K. It would help to state how lambda_vid is set and whether the CLM objective is applied to all description tokens or only caption tokens, since this affects reproducibility of Stage 2/3 training.

Circularity Check

1 steps flagged

The 'emergent interaction deduction' result is circular: the narratives used as deduction inputs are generated by embedding the very interaction labels the LLM is then said to deduce (Φ: (s,v)→N_dense; A.2 recovers Ŷinter from those texts).

specific steps
  1. self definitional [§3.2 (Unified Generation Pipeline, Semantic Expansion) + Appendix A.2 (Zero-Shot Text-Only Deduction; Eqs. 14–15, Table 7)]
    "Φ: (s, v)→ Ndense, where a sparse interaction tags serves as a semantic anchor rooted in the visual context v. ... By embedding the rigid interaction label into this continuous, high-dimensional context, Φ transforms isolated tags into cohesive, visually grounded narratives. ... [A.2] P=Concat(Itask,Tenv,TA,TB) (14) ... predicted interaction category Ŷinter purely through text-based logical deduction without any task-specific training."

    The input texts Tenv, TA, TB of the zero-shot deduction experiment are constructed by the Semantic Expansion mapping Φ, which explicitly conditions on and embeds the interaction tag s into the narrative. Therefore Mθ(P)=Ŷinter is not emergent reasoning from independent visual observations; it is label retrieval from text that was written from that label. The paper's conclusion that interactions 'naturally emerge' and that 'direct cognitive reasoning is more effective than cumbersome explicit visual modeling' is forced by construction: the ground-truth interaction category is the source of the narrative, not a deduction from it. Table 7's comparison against explicit fusion therefore measures whether the caption contains its own tag, not a learned understanding of social interactions.

full rationale

The geometric tracking evaluation (HOTA/IDF1/TETA on BenSMOT and TAO splits, standard detectors) is independent and self-contained: LLMTrack's tracking scores are computed against established benchmarks with conventional metrics, and the improvements there do not depend on the semantic protocol. The dataset construction (Qwen3-VL generation with human-in-the-loop checks) is also transparently described. The formal circularity is localized to the 'emergent deduction' claim in the Introduction/Abstract and Appendix A.2: because the dense narratives used as deduction inputs were generated from the interaction tags that serve as deduction targets, the F1 gains are an artifact of label embedding, not evidence of emergence. I also note, without counting it as definitional circularity, that Appendix G.2 explicitly masks undetected objects and takes the maximum caption score over ID-switched fragments; the paper discloses this decoupling, but it means Table 3's 'qualitative leap' is a best-fragment, detected-only conditional captioning score rather than an end-to-end semantic-reasoning measure. The LLM-generated ground truth and LLM-as-judge evaluation further reduce external independence, but those are benchmark-validity concerns rather than by-construction reductions. Overall: one load-bearing result reduces by construction, while the tracking benchmark and framework retain independent content, giving a partial circularity score of 6.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The paper contributes no new physical or theoretical entities; its additional structure is empirical. The main free parameters are thresholds, window size, and evaluation weights chosen by pilot experiments or ablations. The central unstated assumptions are that LLM-generated text is valid ground truth and that semantic ability can be evaluated after masking tracking failures.

free parameters (4)
  • τ = 7.0 alignment threshold = 7.0 (1-10 scale)
    Set via a 100-video pilot to flag >90% of severe temporal hallucinations; determines which ~5% of sequences receive manual correction (§3.3, Appendix E.3).
  • θ_score, θ_IoU assignment thresholds = not reported
    Used in Eq. 19 to assign extrapolated ground truth on unannotated TAO frames and to model disappearance/drift; values are not given, so replication requires guessing.
  • Fusion window L = 30 frames
    Chosen from an ablation on BenSMOT (Table 6); semantic metrics improve up to L=30, while the offline L=Full upper bound is still higher.
  • Final Score weights 0.7·S_text + 0.3·S_fact = 0.7 / 0.3
    Used to select Qwen3-VL-32B as the generation engine (Eq. 25); the weights are chosen without reported justification.
axioms (3)
  • domain assumption Interaction is a natural logical deduction from individual behaviors and environmental context, not a recognition target requiring explicit supervision.
    Stated in §1 and Appendix A.2; motivates dropping the explicit interaction branch. If false, the framework lacks the relational modeling needed for SMOT.
  • domain assumption LLM-generated captions from Qwen3-VL-32B are sufficiently faithful to serve as ground truth for training and evaluating semantic tracking.
    Dense captions are produced by a generative MLLM and only 255 flagged videos were manually corrected (§3.3). Semantic evaluation compares model outputs to these LLM-generated references.
  • domain assumption Grounding DINO + ByteTrack provide trajectories reliable enough for semantic reasoning, and semantic errors caused by tracking failures can be excluded from semantic metrics.
    Appendix G.1 admits semantic reasoning depends on upstream tracking; Appendix G.2 then masks tracking failures in caption scoring, assuming semantic ability can be isolated.

pith-pipeline@v1.3.0-alltime-deepseek · 42069 in / 12801 out tokens · 128041 ms · 2026-08-03T11:21:50.006315+00:00 · methodology

0 comments
read the original abstract

Semantic Multi-Object Tracking (SMOT) is evolving from purely geometric localization toward comprehensive video understanding. However, existing paradigms predominantly rely on closed-set interaction tags and fragmented perception pipelines, creating a bottleneck that prevents the full utilization of Multi-modal Large Language Models (MLLMs) for dynamic scenes. In this paper, we elevate SMOT from rigid classification to an open-ended generative reasoning task. To support this paradigm shift, we introduce Grand-SMOT, a large-scale benchmark featuring high-density, dual-stream narratives. This dataset explicitly decouples micro-level individual dynamics from macro-level environmental contexts, directly resolving the semantic scarcity of prior tracking datasets. Furthermore, we propose LLMTrack, a unified MLLM-driven framework for dynamic SMOT. Guided by a verifiable ``\textit{Macro-Understanding-First}'' mechanism, LLMTrack employs a Spatio-Temporal Fusion Module to compress discrete geometric trajectories into continuous semantic tokens, effectively suppressing temporal hallucinations in long-sequence tracking. Extensive experiments, utilizing a novel decoupled evaluation protocol, validate that LLMTrack achieves state-of-the-art geometric tracking robustness while delivering a qualitative leap in generative semantic reasoning. The code and datasets are publicly available at https://github.com/liaopan-lp/LLMTrack-GrandSMOT.

Figures

Figures reproduced from arXiv: 2601.06550 by Dingwen Zhang, Di Wu, Feng Yang, Jinwen Yu, Pan Liao, Wang Zhao.

Figure 1
Figure 1. Figure 1: Statistical Comparison. We vi￾sualize the data distribution of our Grand￾SMOT against BenSMOT [22]. Grand￾SMOT achieves a superior balance in se￾quence length, instance density, and seman￾tic richness. The efficacy of large models is inherently data-driven [11, 16]; yet, the current SMOT landscape suf￾fers from a structural deficiency in high-quality instruction-tuning data. Existing datasets fail to meet … view at source ↗
Figure 2
Figure 2. Figure 2: Our pipeline uses Expansion (top) to upgrade sparse labels to rich descrip￾tions, and Generation (bottom) to synthesize video and instance dynamics via a segment-to-global approach. evolution and fine-grained micro-actions. This dual-stream structure compels the tracker to implicitly reason about relations, rather than merely memorizing biased label co-occurrences. 3.2 Unified Generation Pipeline Creating … view at source ↗
Figure 3
Figure 3. Figure 3: LLMTrack architecture. Online processing uses backbone visual features and tracker instance features in a Fusion Module (Video Fusion for long-term context, Instance Fusion for short-term dynamics). Fused tokens + previous-frame description enable LLM to generate current-frame narratives via Cross-Attention (CA), which integrates features during single-object captioning. 4.1 Overview We formulate SMOT as a… view at source ↗
Figure 4
Figure 4. Figure 4: The progressive three-stage training paradigm. Red indicates train￾able components, while blue indicates frozen ones. Step 1 focuses on tracking. Step 2 aligns fusion with LLM. Step 3 fine-tunes LLM via LoRA [15]. Training a unified architecture for both geometric tracking and seman￾tic reasoning is challenging. To ensure stability and efficiency, we propose a progressive training strategy. Stage 1: Geomet… view at source ↗
Figure 5
Figure 5. Figure 5: The detailed architecture of our early explicit feature fusion design. The bot￾tom panel illustrates the hierarchical extraction of instance-level trajectories, explicitly fused interaction-level features (via concatenation and MLP), and video-level contex￾tual embeddings. The top panel details the subsequent integration within the LLM: tokenized multi-modal features first undergo global Self-Attention, fo… view at source ↗
Figure 6
Figure 6. Figure 6: Illustration of the sparse frame sampling strategy during the initial geomet￾ric tracking stage. To mitigate the computational overhead of full temporal back￾propagation, gradients are only computed for a sparse subset of sampled frames (orange blocks with fire icons), while the remaining frames only perform forward passes without gradient tracking (blue blocks with snowflake icons). Crucially, as establis… view at source ↗
Figure 7
Figure 7. Figure 7: Comprehensive Statistics of the Grand-SMOT Benchmark. [PITH_FULL_IMAGE:figures/full_fig_p029_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative Result from the BenSMOT Dataset (Example 1). [PITH_FULL_IMAGE:figures/full_fig_p038_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative Result from the BenSMOT Dataset (Example 2). [PITH_FULL_IMAGE:figures/full_fig_p039_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative Result from the TAO Dataset. [PITH_FULL_IMAGE:figures/full_fig_p040_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Evaluation Strategy for ID Switches. When a single ground-truth tra￾jectory is fragmented into multiple predicted tracklets (e.g., Tracked ID 3 and ID 6), both predicted captions are evaluated against the GT caption. The maximum score (e.g., 3.4 > 3.1 → 3.4) is assigned as the final metric for the GT instance, decoupling the semantic reasoning assessment from temporal association errors. 2. Masking Missin… view at source ↗
Figure 12
Figure 12. Figure 12: Evaluation Strategy for Missing Tracking. [PITH_FULL_IMAGE:figures/full_fig_p043_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

    cs.CV 2026-06 unverdicted novelty 6.0

    A language dependency parsing mechanism combined with Qwen-VL enables adaptive updates to textual descriptions for improved vision-language tracking performance on benchmarks like TNL2K and LaSOT.

Reference graph

Works this paper leans on

87 extracted references · 16 linked inside Pith · cited by 1 Pith paper

  1. [1]

    arXiv preprint arXiv:2509.23661 (2025) 2, 3, 4, 11, 24, 35, 36

    An, X., Xie, Y., Yang, K., Zhang, W., Zhao, X., Cheng, Z., Wang, Y., Xu, S., Chen, C., Wu, C., et al.: Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661 (2025) 2, 3, 4, 11, 24, 35, 36

    Pith/arXiv arXiv 2025

  2. [2]

    In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summa- rization

    Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with im- proved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summa- rization. pp. 65–72 (2005) 11

    2005

  3. [3]

    In: 2016 IEEE International Conference on Image Processing (ICIP)

    Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. In: 2016 IEEE International Conference on Image Processing (ICIP). pp. 3464–3468 (Sep 2016) 3, 12, 13, 30, 31, 32, 33, 34

    2016

  4. [4]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Cao, J., Pang, J., Weng, X., Khirodkar, R., Kitani, K.: Observation-centric sort: Rethinking sort for robust multi-object tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9686–9696 (2023) 12, 13, 30, 31, 32, 33, 34

    2023

  5. [5]

    arXiv preprint arXiv:2402.11684 (2024) 3

    Chen,G.H.,Chen,S.,Zhang,R.,Chen,J.,Wu,X.,Zhang,Z.,Chen,Z.,Li,J.,Wan, X., Wang, B.: Allava: Harnessing gpt4v-synthesized data for lite vision-language models. arXiv preprint arXiv:2402.11684 (2024) 3

    Pith/arXiv arXiv 2024

  6. [6]

    In: Eu- ropean Conference on Computer Vision

    Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., Lin, D.: Sharegpt4v: Improving large multi-modal models with better captions. In: Eu- ropean Conference on Computer Vision. pp. 370–387. Springer (2024) 2, 3, 4

    2024

  7. [7]

    In: European conference on computer vision

    Dave, A., Khurana, T., Tokmakov, P., Schmid, C., Ramanan, D.: Tao: A large-scale benchmark for tracking any object. In: European conference on computer vision. pp. 436–454. Springer (2020) 2, 4, 5, 6, 10, 12

    2020

  8. [8]

    arXiv preprint arXiv:2003.09003 (2020) 6

    Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid, I., Roth, S., Schindler, K., Leal-Taixé, L.: Mot20: A benchmark for multi object tracking in crowded scenes. arXiv preprint arXiv:2003.09003 (2020) 6

    Pith/arXiv arXiv 2003

  9. [9]

    arXiv preprint arXiv:1810.04805 (2018) 4, 11

    Devlin,J.,Chang,M.W.,Lee,K.,Toutanova,K.:Bert:Pre-trainingofdeepbidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) 4, 11

    Pith/arXiv arXiv 2018

  10. [10]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Fu, S., Yang, Q., Mo, Q., Yan, J., Wei, X., Meng, J., Xie, X., Zheng, W.S.: Llmdet: Learning strong open-vocabulary object detectors under the supervision of large language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 14987–14997 (2025) 4, 7

    2025

  11. [11]

    Advances in Neural Information Processing Systems36, 27092–27112 (2023) 2

    Gadre, S.Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al.: Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems36, 27092–27112 (2023) 2

    2023

  12. [12]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Gao, R., Qi, J., Wang, L.: Multiple object tracking as id prediction. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 27883–27893 (2025) 1, 3, 10, 25

    2025

  13. [13]

    arXiv preprint arXiv:2107.08430 (2021) 12

    Ge, Z.: Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430 (2021) 12

    Pith/arXiv arXiv 2021

  14. [14]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Gupta, A., Dollar, P., Girshick, R.: Lvis: A dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5356–5364 (2019) 4

    2019

  15. [15]

    In: ICLR

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. In: ICLR. vol. 1, p. 3 (2022) 9 16 F. Author et al

    2022

  16. [16]

    arXiv preprint arXiv:2001.08361 (2020) 2

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020) 2

    Pith/arXiv arXiv 2001

  17. [17]

    arXiv preprint arXiv:2408.03326 (2024) 2, 3, 4, 11, 24, 35, 36

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024) 2, 3, 4, 11, 24, 35, 36

    Pith/arXiv arXiv 2024

  18. [18]

    arXiv preprint arXiv:2503.10616 (2025) 12, 13, 31, 34

    Li,J.,Yu,E.,Chen,S.,Tao,W.:Ovtr:End-to-endopen-vocabularymultipleobject tracking with transformer. arXiv preprint arXiv:2503.10616 (2025) 12, 13, 31, 34

    Pith/arXiv arXiv 2025

  19. [19]

    Science China Information Sciences 68(10), 200102 (2025) 4

    Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y.: Videochat: Chat-centric video understanding. Science China Information Sciences 68(10), 200102 (2025) 4

    2025

  20. [20]

    In: European conference on computer vision

    Li, S., Danelljan, M., Ding, H., Huang, T.E., Yu, F.: Tracking every thing in the wild. In: European conference on computer vision. pp. 498–515. Springer (2022) 11

    2022

  21. [21]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Li, S., Fischer, T., Ke, L., Ding, H., Danelljan, M., Yu, F.: Ovtrack: Open- vocabulary multiple object tracking. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5567–5577 (2023) 12, 13, 30, 31, 34

    2023

  22. [22]

    In: European Conference on Computer Vision

    Li, Y., Li, Q., Wang, H., Ma, X., Yao, J., Dong, S., Fan, H., Zhang, L.: Beyond mot: Semantic multi-object tracking. In: European Conference on Computer Vision. pp. 276–293. Springer (2024) 1, 2, 4, 5, 6, 11, 12, 13, 24, 32, 33, 34

    2024

  23. [23]

    IEEE Transactions on Industrial Informatics pp

    Liao, P., Yang, F., Wu, D., Yu, J., Li, X., Zhang, D.: Fasttracktr: Real-time multi- object tracking with transformers for real world. IEEE Transactions on Industrial Informatics pp. 1–11 (2025) 1

    2025

  24. [24]

    IEEE Transactions on Industrial Informatics (2025) 7, 10

    Liao, P., Yang, F., Wu, D., Yu, J., Li, X., Zhang, D.: Fasttracktr: Real-time multi- object tracking with transformers for real world. IEEE Transactions on Industrial Informatics (2025) 7, 10

    2025

  25. [25]

    Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection.In:ProceedingsoftheIEEEinternationalconferenceoncomputervision. pp. 2980–2988 (2017) 9

    2017

  26. [26]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tun- ing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26296–26306 (2024) 4

    2024

  27. [27]

    In: European conference on computer vision

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In: European conference on computer vision. pp. 38–55. Springer (2024) 3, 7, 11, 26, 31

    2024

  28. [28]

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer:Hierarchical visiontransformerusingshiftedwindows.In:Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021) 11

    2021

  29. [29]

    International journal of computer vision129, 548–578 (2021) 11

    Luiten, J., Osep, A., Dendorfer, P., Torr, P., Geiger, A., Leal-Taixé, L., Leibe, B.: Hota: A higher order metric for evaluating multi-object tracking. International journal of computer vision129, 548–578 (2021) 11

    2021

  30. [30]

    arXiv preprint arXiv:2511.17681 (2025) 1, 4

    Lv, W., Zhang, N., Sun, H., Jiang, H., Zhao, K., Xiao, J., Zeng, D.: Vision-motion- reference alignment for referring multi-object tracking via multi-modal large lan- guage models. arXiv preprint arXiv:2511.17681 (2025) 1, 4

    arXiv 2025

  31. [31]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Maaz, M., Rasheed, H., Khan, S., Khan, F.: Video-chatgpt: Towards detailed video understanding via large vision and language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12585–12602 (2024) 11, 32 LLMTrack 17

    2024

  32. [32]

    In: 2023 IEEE International conference on image processing (ICIP)

    Maggiolino, G., Ahmad, A., Cao, J., Kitani, K.: Deep oc-sort: Multi-pedestrian tracking by adaptive re-identification. In: 2023 IEEE International conference on image processing (ICIP). pp. 3025–3029. IEEE (2023) 1, 4

    2023

  33. [33]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Meinhardt, T., Kirillov, A., Leal-Taixe, L., Feichtenhofer, C.: Trackformer: Multi- object tracking with transformers. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8844–8854 (2022) 3

    2022

  34. [34]

    Milan, A., Leal-Taixe, L., Reid, I., Roth, S., Schindler, K.: MOT16: A Benchmark for Multi-Object Tracking (May 2016) 6

    2016

  35. [35]

    Advances in Neural Information Processing Systems 36, 3205–3219 (2023) 6

    Nguyen, P., Quach, K.G., Kitani, K., Luu, K.: Type-to-track: Retrieve any object via prompt-based tracking. Advances in Neural Information Processing Systems 36, 3205–3219 (2023) 6

    2023

  36. [36]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Pang, J., Qiu, L., Li, X., Chen, H., Li, Q., Darrell, T., Yu, F.: Quasi-dense similarity learning for multiple object tracking. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 164–173 (2021) 3

    2021

  37. [37]

    In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311–318 (2002) 11

    2002

  38. [38]

    Advances in neural information processing sys- tems32(2019) 11

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing sys- tems32(2019) 11

    2019

  39. [39]

    Available at SSRN 5541079 (2025) 1

    Raja, R., Vats, A., Thawakar, O., Ashraf, T.: Object tracking: A comprehensive survey from classical approaches to large vision-language and foundation models. Available at SSRN 5541079 (2025) 1

    2025

  40. [40]

    IEEE transactions on pattern analysis and machine intelligence39(6), 1137–1149 (2016) 9, 12

    Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object de- tection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence39(6), 1137–1149 (2016) 9, 12

    2016

  41. [41]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition

    Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: General- ized intersection over union: A metric and a loss for bounding box regression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 658–666 (2019) 9

    2019

  42. [42]

    In: European conference on computer vision

    Ristani, E., Solera, F., Zou, R., Cucchiara, R., Tomasi, C.: Performance measures and a data set for multi-target, multi-camera tracking. In: European conference on computer vision. pp. 17–35. Springer (2016) 11

    2016

  43. [43]

    In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Sun, P., Cao, J., Jiang, Y., Yuan, Z., Bai, S., Kitani, K., Luo, P.: DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 20961–20970. IEEE, New Orleans, LA, USA (Jun 2022) 6

    2022

  44. [44]

    arXiv preprint arXiv:2012.15460 (2020) 24, 30, 32, 33

    Sun, P., Cao, J., Jiang, Y., Zhang, R., Xie, E., Yuan, Z., Wang, C., Luo, P.: Transtrack: Multiple object tracking with transformer. arXiv preprint arXiv:2012.15460 (2020) 24, 30, 32, 33

    Pith/arXiv arXiv 2012

  45. [45]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Sun, Y., Cheng, C., Zhang, Y., Zhang, C., Zheng, L., Wang, Z., Wei, Y.: Circle loss: A unified perspective of pair similarity optimization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6398–6407 (2020) 9

    2020

  46. [46]

    arXiv preprint arXiv:2403.05530 (2024) 2, 4, 33 18 F

    Team, G., Georgiev, P., Lei, V.I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vin- cent,D.,Pan,Z.,Wang,S.,etal.:Gemini1.5:Unlockingmultimodalunderstanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024) 2, 4, 33 18 F. Author et al

    Pith/arXiv arXiv 2024

  47. [47]

    arXiv preprint arXiv:2506.07900 (2025) 6, 35, 36

    Team, M., Xiao, C., Li, Y., Han, X., Bai, Y., Cai, J., Chen, H., Chen, W., Cong, X., Cui, G., et al.: Minicpm4: Ultra-efficient llms on end devices. arXiv preprint arXiv:2506.07900 (2025) 6, 35, 36

    Pith/arXiv arXiv 2025

  48. [48]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4566–4575 (2015) 11

    2015

  49. [49]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, N., Zhu, G., Li, H., Zhang, L., Shah, S.A.A., Bennamoun, M.: Lan- guage model guided interpretable video action reasoning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18878– 18887 (2024) 1, 4

    2024

  50. [50]

    arXiv preprint arXiv:2409.12191 (2024) 2, 11, 35, 36

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024) 2, 11, 35, 36

    Pith/arXiv arXiv 2024

  51. [51]

    In: The Thirty-ninth Annual Conference on Neural Informa- tion Processing Systems 2

    Wang,Q.,Chen,S.,Shen,Y.:Causalvtg:Towardsrobustvideotemporalgrounding via causal inference. In: The Thirty-ninth Annual Conference on Neural Informa- tion Processing Systems 2

  52. [52]

    arXiv preprint arXiv:2508.18265 (2025) 35, 36

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al.: Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 (2025) 35, 36

    Pith/arXiv arXiv 2025

  53. [53]

    Wang, Z., Zheng, L., Liu, Y., Li, Y., Wang, S.: Towards real-time multi-object tracking.In:Europeanconferenceoncomputervision.pp.107–122.Springer(2020) 7

    2020

  54. [54]

    arXiv preprint arXiv:2206.07682 (2022) 2

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.: Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022) 2

    Pith/arXiv arXiv 2022

  55. [55]

    In: European Conference on Computer Vision

    Weng, Y., Han, M., He, H., Chang, X., Zhuang, B.: Longvlm: Efficient long video understanding via large language models. In: European Conference on Computer Vision. pp. 453–470. Springer (2024) 10

    2024

  56. [56]

    Neural computation1(2), 270–280 (1989) 26

    Williams, R.J., Zipser, D.: A learning algorithm for continually running fully re- current neural networks. Neural computation1(2), 270–280 (1989) 26

    1989

  57. [57]

    In: 2017 IEEE international conference on image processing (ICIP)

    Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric. In: 2017 IEEE international conference on image processing (ICIP). pp. 3645–3649. IEEE (2017) 3, 12, 13, 24, 30, 31, 32, 33, 34

    2017

  58. [58]

    In: Proceedings of the ieee/cvf conference on computer vision and pattern recognition

    Wu, C.Y., Li, Y., Mangalam, K., Fan, H., Xiong, B., Malik, J., Feichtenhofer, C.: Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In: Proceedings of the ieee/cvf conference on computer vision and pattern recognition. pp. 13587–13597 (2022) 10

    2022

  59. [59]

    arXiv preprint arXiv:2505.09388 (2025) 2, 3, 5, 11, 35, 36

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025) 2, 3, 5, 11, 35, 36

    Pith/arXiv arXiv 2025

  60. [60]

    In: European conference on computer vision

    Yang, J., Dong, Y., Liu, S., Li, B., Wang, Z., Tan, H., Jiang, C., Kang, J., Zhang, Y., Zhou, K., et al.: Octopus: Embodied vision-language programmer from environ- mental feedback. In: European conference on computer vision. pp. 20–38. Springer (2024) 1

    2024

  61. [61]

    In: European Conference on Computer Vision

    Yin,W.,Cai,Z.,Wang,R.,Wang,F.,Wei,C.,Mei,H.,Xiao,W.,Yang,Z.,Sun,Q., Yamashita, A., et al.: Whac: World-grounded humans and cameras. In: European Conference on Computer Vision. pp. 20–37. Springer (2024) 1

    2024

  62. [62]

    arXiv preprint arXiv:2602.15763 (2026) 33 LLMTrack 19

    Zeng, A., Lv, X., Hou, Z., Du, Z., Zheng, Q., Chen, B., Yin, D., Ge, C., Xie, C., Wang, C., et al.: Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763 (2026) 33 LLMTrack 19

    Pith/arXiv arXiv 2026

  63. [63]

    In: European Conference on Computer Vision

    Zeng, F., Dong, B., Zhang, Y., Wang, T., Zhang, X., Wei, Y.: Motr: End-to-end multiple-object tracking with transformer. In: European Conference on Computer Vision. pp. 659–675. Springer (2022) 1, 3, 30, 32, 33

    2022

  64. [64]

    In: Proceedings of the 33rd ACM International Confer- ence on Multimedia

    Zhang, S., Hao, X., Tang, Y., Zhang, L., Wang, P., Wang, Z., Ma, H., Zhang, S.: Video-cot: A comprehensive dataset for spatiotemporal understanding of videos based on chain-of-thought. In: Proceedings of the 33rd ACM International Confer- ence on Multimedia. pp. 12745–12752 (2025) 2, 4

    2025

  65. [65]

    In: European conference on computer vision

    Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., Wang, X.: Bytetrack: Multi-object tracking by associating every detection box. In: European conference on computer vision. pp. 1–21. Springer (2022) 1, 3, 8, 12, 13, 24, 30, 31, 32, 33, 34

    2022

  66. [66]

    arXiv preprint arXiv:2510.13235 (2025) 1, 4

    Zhang, Y., Zhao, J., Nie, S., Kuang, J., Wang, S.: Epiptrack: Rethinking prompt modeling with explicit and implicit prompts for multi-object tracking. arXiv preprint arXiv:2510.13235 (2025) 1, 4

    arXiv 2025

  67. [67]

    interaction

    Zhou, X., Yin, T., Koltun, V., Krähenbühl, P.: Global tracking transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. pp. 8771–8780 (2022) 12, 13, 31, 34 20 F. Author et al. Appendix of LLMTrack A Explicit Interaction Modeling and Emergent Deduction ............. 20 A.1 Baseline Architecture: Explicit Featur...

    arXiv 2022

  68. [68]

    For example, a single GT cyclist (ID 73) might be predicted as two separate tracked IDs (ID 3 and ID 6)

    Mitigating the Impact of ID Switches (Fragmented Trajectories) As illustrated in Figure 11, an ID switch occurs when a single Ground Truth (GT) target is fragmented into multiple predicted tracklets due to occlusion or tracking drift. For example, a single GT cyclist (ID 73) might be predicted as two separate tracked IDs (ID 3 and ID 6). To fairly assess ...

  69. [69]

    Under standard protocols, the absence of a prediction would result in a zero score for the corresponding GT caption, dragging down the overall semantic average

    Masking Missing Tracks (Undetected Objects) As shown in Figure 12, severe visual occlusion or extreme scale variation can cause an object to be entirely missed by the detector (Detection Score= 0). Under standard protocols, the absence of a prediction would result in a zero score for the corresponding GT caption, dragging down the overall semantic average...

  70. [70]

    Detailed environment and setting (location type, weather/lighting, background elements)

  71. [71]

    Appearance details of people/animals (clothing, physical features, posture)

  72. [72]

    Specific actions and their sequence (what happens step by step)

  73. [73]

    Object details and interactions (what objects are involved, how they are used)

  74. [74]

    Based on the provided video frames and the orig- inal instance caption, generate a more detailed description for this specific instance (object/person/animal)

    Spatial relationships (where things are positioned relative to each other) Requirements: – Output ONLY the expanded description in English – Keep the description factual and objective (describe what you see, not interpre- tations) – Be detailed but concise (2-4 sentences) – Do not include any preamble or explanation, just the description [Instance Caption...

  75. [75]

    Detailed appearance (color, size, clothing/fur pattern, distinguishing features)

  76. [76]

    Position and movement in the scene (where they are, how they move)

  77. [77]

    Interactions with other objects/instances (what they do with others)

  78. [78]

    Temporal changes throughout the video (how their state/position changes)

  79. [79]

    appears to

    Any distinctive characteristics or behaviors Requirements: – Output ONLY the expanded description in English – Keep the description factual and objective – Be detailed but concise (2-4 sentences) – Do not include any preamble or explanation, just the description Prompt for Dataset Generation (TAO): [Video-Level Caption] You are an expert video annotator f...

  80. [80]

    Do not browse the web or read external files

    Only judge based on the provided text input. Do not browse the web or read external files

Showing first 80 references.