Pith. sign in

REVIEW 3 major objections 6 minor 52 references

ReferTrack claims that grounding target identification in explicit image-space bounding boxes—selecting one indexed detection before planning—makes a single-camera agent match or beat multi-camera policies on identification-heavy tracking.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:54 UTC pith:VGAUDABA

load-bearing objection A well-built empirical extension that makes target identification an explicit indexed-bbox selection, with real gains on EVT-Bench; but the unmeasured top-K truncation/detector recall and single-run reporting keep me from fully endorsing the strong claims. the 3 major comments →

arxiv 2607.20061 v1 pith:VGAUDABA submitted 2026-07-22 cs.RO

ReferTrack: Referring Then Tracking for Embodied Visual Tracking

Hanjing Ye , Tianle Zeng , Jiazhao Zhang , Shaoan Wang , Zibo Zhang , Weisi Situ , Yuchen Zhou , Yonggen Ling
show 1 more author
Hong Zhang
This is my paper
classification cs.RO
keywords embodied visual trackingvision-language-action modelreferring expression comprehensionbounding-box catalogtemporal-viewpoint-bbox indicatorsingle-camera trackingtarget identificationsim-to-real
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ReferTrack claims that embodied visual tracking is better when target identification is treated as picking one entry from a set of detected image-space bounding boxes rather than as an abstract spatial reasoning step. The model first emits a single Refer-CoT token that names the indexed pedestrian it will follow (or a fixed ⟨NO EXIST⟩ token), then predicts movement waypoints conditioned on that choice. On the paper's single-camera benchmark, it reports success rates of 89.4% on single-target tracking, 73.3% on distracted tracking, and 74.1% on ambiguity tracking — improving on prior single-view policies by large margins and matching or exceeding several multi-camera systems on the identification-heavy splits. The authors argue that explicit image-space referring can compensate for limited camera coverage and for the lack of reinforcement-learning fine-tuning. If correct, single-camera, purely supervised policies become a competitive and much simpler recipe for language-guided person following.

Core claim

ReferTrack formulates embodied visual tracking as a two-stage operation over a discrete candidate set. At each timestep, detections from the forward camera are organized into an indexed catalog of pedestrian bounding boxes. The policy's first reasoning action is a single Refer-CoT token selecting the catalog entry that matches the language instruction (or ⟨NO EXIST⟩ if the target is absent); only then does it decode waypoints, conditioned on that selection. The selected bounding box is appended to a sliding-window queue and injected into the visual history of subsequent frames through temporal-viewpoint-bbox indicator (TVBI) tokens, giving the planner target-specific geometric memory. Co-tra

What carries the argument

The load-bearing mechanism is the indexed candidate catalog plus a single Refer-CoT token. An off-the-shelf detector and multi-object tracker produce a set of pedestrian bounding boxes; each box is assigned a special token (⟨ped_k⟩) and its coordinates are projected into the language model's embedding space. The first LLM forward pass performs a classification over these tokens (plus ⟨NO EXIST⟩), making target identification a compact, directly supervised multiple-choice decision in image space rather than an open-ended spatial abstraction. The second mechanism is the temporal-viewpoint-bbox indicator (TVBI) token: the selected target's past bounding boxes, embedded with the same projector,

Load-bearing premise

The central result depends on the off-the-shelf detector and tracker finding the instructed target and keeping it among the top-K largest bounding boxes in the current forward view; the paper does not measure how often the target is absent from the candidate catalog after that truncation.

What would settle it

On the benchmark's distracted and ambiguity splits, compute the fraction of episodes where the instructed target is not present in the detected candidate catalog (missed by the detector/tracker, or cut by the top-K area truncation). If that fraction is substantial and correlates with failures, the reported gains are partly an unmeasured detection bottleneck; if it is near zero, the referring mechanism is confirmed as the decisive factor.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, a single forward-facing camera suffices for embodied tracking; multi-camera setups are not required for identification-heavy scenarios.
  • Explicit image-space referring can substitute for expensive RL fine-tuning: the reported gains come from supervised fine-tuning with a 1:1 mix of navigation and referring-QA data.
  • The candidate-catalog interface decouples identification from planning, so improvements in referring and grounding models can transfer to tracking without new closed-loop navigation data.
  • The TVBI bbox-queue memory is a direct cause of stability: removing both Refer-CoT and TVBI drops success from 73.3% to 55.7% on the distracted split, showing target selection and temporal bbox memory are complementary.
  • The referring-QA co-training recipe suggests a scalable data path: large existing referring and grounding datasets can be converted into tracking supervision through the same indexed-box interface.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report how often the instructed target is missing from the detected candidate catalog after the top-K truncation; if the detector fails or the area-based cut removes the target, the Refer-CoT token cannot select it and the episode cannot succeed. An oracle-catalog experiment would isolate how much of the reported gain depends on raw detection recall.
  • The oracle TVBI ablation (ground-truth bboxes, 81.5% SR on distracted tracking) still trails the expert policy (85.1%) and beats the full model (73.3%), so identification errors, not planning, are the main residual bottleneck; coupling ReferTrack with a re-identification module that can revisit past choices could close that gap.
  • Because the design replaces abstract latent reasoning with a discrete selection, the same interface could be applied to other 'find X, then act' VLA tasks, such as following a particular vehicle or heading to a described object in a scene.
  • The real-world deployments are qualitative demonstrations on a remote server at 10.6 Hz with a forward-facing camera; a quantitative field study with measured target-loss rates under occlusion and network latency would test the sim-to-real claim beyond demonstrations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces ReferTrack, a 4B-parameter vision-language-action (VLA) policy for embodied visual tracking from a single forward-facing camera. The core idea is to formulate target identification as a constrained selection among an indexed set of image-space pedestrian bounding boxes, produced by YOLO11+ByteTrack and truncated to a top-K catalog. A single "Refer-CoT" token selects the target (or outputs NO EXIST), and the selected bbox is pushed into a FIFO queue whose geometric features are injected into historical visual tokens via temporal-viewpoint-bbox indicator (TVBI) tokens. The policy is trained by SFT on 1.3M navigation samples plus 1.3M synthetically generated Refer-QA samples, with losses for trajectory MSE, Refer-CoT cross-entropy, and text cross-entropy. On EVT-Bench in a single-view protocol, the paper reports state-of-the-art success rates of 89.4/73.3/74.1 on the single-target, distracted, and ambiguity splits, and shows qualitative real-robot deployment on a quadruped and a humanoid. The main claim is that explicit image-space referring can compensate for limited camera coverage and reduce the need for larger models or RL fine-tuning.

Significance. If the reported results are reproducible, the paper makes a useful conceptual contribution: it replaces abstract spatial CoT tokens with a grounded, index-based referring step, and shows that this interface can be trained jointly with navigation data and synthetic referring QA data. The architecture is described in enough detail to be internally coherent, the ablation in Table 2 isolates the contributions of Refer-CoT and TVBI, and the oracle variant provides a useful upper bound. The code release, the real-robot validation, and the explicit treatment of the target-absent case via a NO EXIST slot are also strengths. However, as detailed below, the central claim rests on an unmeasured external dependency — the candidate catalog's recall and truncation — and on several claims that lack supporting ablations or statistical grounding. The significance would be higher if these issues were addressed.

major comments (3)
  1. [§3.2 Candidate Catalog; Table 1] The central success claim depends on the instructed target being present in the indexed catalog. §3.2 truncates detections to the top-K bboxes by area, but K is never reported and the recall of YOLO11+ByteTrack+top-K against the ground-truth target on EVT-Bench is never measured. If the target is missed or pruned, the Refer-CoT token cannot select it and the episode cannot succeed, regardless of referring or planning quality. The oracle ablation in Table 2 (TVBI w/ GT bbox) bypasses both the detector and the truncation, so the gap from 81.5 to 73.3 SR conflates detector/pruning failures with the model's own referring errors. Please report K, measure the recall of the candidate catalog on the evaluation splits, and add an ablation that feeds ground-truth detection boxes (or a recall-controlled catalog) while still requiring Refer-CoT selection, to separate detection/truncation from referr
  2. [§3.3 Training; Table 2] The paper claims that co-training on the custom Refer-QA dataset 'strengthens' and 'transfers seamlessly' to online tracking, and Refer-QA constitutes half of the Stage-2 training data. Yet Table 2 contains no ablation that removes Refer-QA while keeping the navigation data fixed. Without such a variant, the contribution of the Refer-QA co-training objective to the reported DT/AT improvements is unsupported. This is load-bearing because the method's stated novelty includes the co-training recipe as a means of improving identification. Please add an ablation such as 'navigation data only, no Refer-QA' with the same model and training budget.
  3. [Table 1; §4.2] All EVT-Bench results are reported as single runs with no error bars, variance, or significance tests. The headline comparisons to TrackVLA++ and VLingNav are absolute differences from single runs, which is risky for a central SOTA claim. Additionally, the abstract and §4.2 state that ReferTrack 'matches or even surpasses several multi-camera baselines' on DT/AT, while Table 1 labels those rows as external references from other publications with 3–4 cameras. This is not a controlled comparison and the wording should be softened or accompanied by a statistical analysis. At minimum, report multiple seeds with mean±std and, if possible, a paired test for the single-view comparisons; for the multi-camera references, state explicitly that input settings differ.
minor comments (6)
  1. [§3.2] K and H are never numerically specified in the main text or Appendix, though the paper says hyperparameters are in the Appendix. Please report K, H, and the queue capacity explicitly in §3.2 or in a reproducibility table in the Appendix.
  2. [Eq. (1)] It would help to clarify whether E_refer^T in the second forward pass is the token embedding of the selected index or the token id itself, and how the action head consumes E_A^T.
  3. [Table 1] NavFoM single-view row has a missing CR entry, and the multi-camera reference rows are not visually distinguished from the single-view methods. Consider separate sub-tables or clearer formatting so the main comparison is unambiguous.
  4. [Figure 1] The example instruction contains 'T-shit' (should be 'T-shirt'); also the catalog in the figure uses inconsistent capitalization (<Ped_8> vs <ped_8>). Please fix.
  5. [§B.1] There is a typo: 'different embodiedments' should be 'different embodiments'.
  6. [§A.2] The referred negative caption is said to be 'not present in the image,' but it is not stated how the negative caption is selected from SYNTH-PEDES or whether it is a random or hard negative. A sentence clarifying this would aid reproducibility.

Circularity Check

0 steps flagged

No construction-level circularity; external EVT-Bench results are measured, and self-citations are lineage only.

full rationale

ReferTrack's reported SR/TR/CR numbers are empirical measurements on the held-out EVT-Bench test splits (Table 1), not quantities derived from model equations or fitted constants, so the central claim is externally falsifiable. The refer-then-track formulation (Eq. 1) requires the LLM to output a discrete bbox-index token before waypoint decoding; success is scored by the downstream Habitat following metrics, so a wrong selection fails the episode and correctness is not by construction. Refer-QA co-training is auxiliary supervision on SYNTH-PEDES composites, and TVBI tokens are an architectural mechanism (Eq. 2) whose effect is measured by ablations (Table 2); neither reduces the benchmark result to its inputs. Self-citations to TrackVLA, TrackVLA++, and NavFoM appear as related work, baseline comparisons, and a training-recipe reference (Appendix A.3); they are not used as uniqueness theorems or as the sole justification for the paradigm. The detector-recall / top-K truncation dependency highlighted by the skeptic is a real external limitation and a possible confound for the 'image-space referring beats camera coverage' interpretation, but it is not a definitional circularity: YOLO11+ByteTrack is an off-the-shelf external component whose misses would only cap measured success, not guarantee it. No equation in the paper makes a reported result equal to an input by construction. Score 2 reflects mild self-referential lineage pressure from the authors' prior EVT work, not a circular derivation.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 4 invented entities

The result depends on detector completeness and top-K catalog truncation (§3.2), on the quality of the custom oracle expert data (Appendix A.1), and on transfer from synthetic Refer-QA (Appendix A.2). Several control hyperparameters (K, H, noise probability, α) are not varied, and their values are partly unreported.

free parameters (5)
  • Trajectory-loss weight α = 10
    Eq. (3) sets α=10; no sensitivity study shows robustness to this choice.
  • Catalog top-K cap K = not reported
    Controls whether the target remains selectable in crowded scenes; value is never given (§3.2).
  • Visual history window H = not reported
    The number of historical frames and queue length (H−1) is central to TVBI; value is not reported.
  • Queue noise injection probability = not reported
    Training occasionally injects wrong indices to simulate tracking errors; probability is not specified (§3.2).
  • Refer-QA composition settings = 2-3 crops, scale 0.75-1.5
    Hand-chosen synthesis parameters in Appendix A.2; no ablation of their effect.
axioms (5)
  • domain assumption YOLO11+ByteTrack detections contain the instructed target and are sorted into a stable index order.
    Candidate catalog C_T is built directly from detector output (§3.2); if the detector misses or reorders the target, the model cannot select the correct index.
  • domain assumption Top-K truncation by bbox area preserves the target when more than K pedestrians are visible.
    §3.2: 'we prioritize the top-K candidates based on bounding box area.' No experiment checks how often this removes the target.
  • domain assumption The custom oracle expert controller generates high-quality, representative following behavior for training.
    Appendix A.1 replaces proprietary curation with a homemade geodesic+PD controller; if its behavior distribution differs from the evaluation distribution, SFT might be biased.
  • domain assumption SYNTH-PEDES-based synthetic Refer-QA examples transfer to Habitat simulation and real robots.
    Appendix A.2 pastes 2-3 captioned pedestrian crops onto backgrounds; the paper relies on this to improve online identification without direct evidence of the transfer.
  • domain assumption A single special-token classification over a dynamically changing catalog is a learnable and reliable interface for a 4B LLM.
    The model must map arbitrary catalog sizes/orderings to the correct special token; this is an implicit capability assumption of the architecture.
invented entities (4)
  • Refer-CoT selection token no independent evidence
    purpose: Single-token intermediate decision selecting one indexed bbox (or NO EXIST) before trajectory decoding.
    Shown to matter by Table 2 ablation (removing Refer-CoT+TVBI drops DT SR 73.3→55.7), but only internal.
  • TVBI tokens no independent evidence
    purpose: Temporal-viewpoint-bbox indicator tokens that inject past selected target bboxes into visual history.
    Ablation w/o TVBI drops DT SR 73.3→70.4; internal evidence only.
  • NO EXIST virtual catalog slot no independent evidence
    purpose: Allows the policy to represent that the target is absent from the current view.
    No separate analysis/report of its accuracy or effect.
  • Referred-target bbox queue no independent evidence
    purpose: FIFO memory of selected target bboxes consumed by TVBI tokens.
    It is the mechanism behind TVBI; no external validation.

pith-pipeline@v1.3.0-alltime-deepseek · 13381 in / 15725 out tokens · 133219 ms · 2026-08-01T10:54:06.835235+00:00 · methodology

0 comments
read the original abstract

Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language using only onboard vision. While recent vision-language-action (VLA) policies unify target identification and trajectory planning, their chain-of-thought (CoT) reasoning often operates in abstract spatial latents that are difficult to supervise and weakly aligned with explicit image-space detections. To address this, we introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera. Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision. To preserve target motion cues over time, ReferTrack maintains a sliding-window queue of previously selected bounding boxes, injecting their geometric features into the visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. We further enhance target identification by co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance with success rates of 89.4%, 73.3%, and 74.1% on the single-target, distracted, and ambiguity tracking splits, respectively -- matching or even surpassing several multi-camera baselines on identification-heavy tasks. Finally, real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities. Code is available at https://github.com/MedlarTea/referTrack.

Figures

Figures reproduced from arXiv: 2607.20061 by Hanjing Ye, Hong Zhang, Jiazhao Zhang, Shaoan Wang, Tianle Zeng, Weisi Situ, Yonggen Ling, Yuchen Zhou, Zibo Zhang.

Figure 1
Figure 1. Figure 1: ReferTrack formulates embodied visual tracking as referring then tracking: forward-view detections are organized as indexed bboxes, and a single Refer-CoT token selects the instructed pedestrian before trajectory prediction. This facilitates image-grounded reasoning within a unified, end-to-end policy. ∗Corresponding authors. (email: rolandling@tencent.com, hzhang@sustech.edu.cn). arXiv:2607.20061v1 [cs.RO… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of ReferTrack. ReferTrack first grounds the language instruction and visual stream by selecting one indexed bbox from the current detections, then predicts tracking waypoints conditioned on this Refer-CoT decision. The selected bbox is stored as target-specific memory and injected into future visual history through TVBI tokens. tokens, catalog, and language tokens are concatenated and passed throu… view at source ↗
Figure 3
Figure 3. Figure 3: Real-world qualitative results. Left: Unitree Go2 follows a target pedestrian through cluttered obstacles under a narrow forward-camera FoV. Right: Unitree G1 maintains the correct referred target under multi-person interference. discrete bbox geometry into continuous tracking policies. More broadly, such target-specific visual guidance could also be supplied by strong tracking or person re-identification … view at source ↗
Figure 4
Figure 4. Figure 4: Expert tracking data examples. Under the instruction “Track the person wearing a red and yellow superhero suit.”, the oracle controller covers diverse following behaviors, including backing up when the target approaches, chasing an intermediate waypoint around large turns, normal rear-side following, and stopping when the target stops (left to right). A.2 Refer-QA Data Generation We formulate auxiliary sup… view at source ↗
Figure 5
Figure 5. Figure 5: Refer-QA examples. We synthesize multi-person referring samples by compositing 2–3 captioned pedestrian crops onto diverse backgrounds and supervising the model to select the queried indexed bbox or ⟨NO EXIST⟩. Example prompt. For the first image in [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Robot platforms. The ReferTrack is deployed on a remote high-performance server. Both robot platforms are equipped with a single forward-facing camera (Intel RealSense D455 camera) and a portable Wi-Fi for communication. Several engineering optimizations are used to reduce latency. First, the model checkpoint is loaded only once on server startup, while each robot connection owns an independent streaming s… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 17 linked inside Pith

  1. [1]

    S. Wang, J. Zhang, M. Li, J. Liu, A. Li, K. Wu, F. Zhong, J. Yu, Z. Zhang, and H. Wang. Trackvla: Embodied visual tracking in the wild. InConference on Robot Learning, pages 4139–4164. PMLR, 2025

    2025

  2. [2]

    J. Liu, Y . Qi, J. Zhang, M. Li, S. Wang, K. Wu, H. Ye, H. Zhang, Z. Chen, F. Zhong, et al. Trackvla++: Unleashing reasoning and memory capabilities in vla models for embodied visual tracking.arXiv preprint arXiv:2510.07134, 2025

    arXiv 2025

  3. [3]

    Zhong, P

    F. Zhong, P. Sun, W. Luo, T. Yan, and Y . Wang. Ad-vat+: An asymmetric dueling mechanism for learning and understanding visual active tracking.IEEE Trans. Pattern Anal. Mach. Intell., 43(5):1467–1482, 2021

    2021

  4. [4]

    Zhong, K

    F. Zhong, K. Wu, H. Ci, C. Wang, and H. Chen. Empowering embodied visual tracking with visual foundation models and offline rl. InEuropean Conference on Computer Vision, pages 139–155. Springer, 2024

    2024

  5. [5]

    H. Sun, K. Zhou, H. Gao, S. Zhang, J. Hu, X. Wen, Q. Ye, and M. Tan. Instance-level visual active tracking with occlusion-aware planning.arXiv preprint arXiv:2604.21453, 2026

    Pith/arXiv arXiv 2026

  6. [6]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, et al. Segment anything. InProceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  7. [7]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Zhang, A

    J. Zhang, A. Li, Y . Qi, M. Li, J. Liu, S. Wang, H. Liu, G. Zhou, Y . Wu, X. Li, et al. Embodied navigation foundation model.arXiv preprint arXiv:2509.12129, 2025

    arXiv 2025

  10. [10]

    Y . Liu, L. Gao, L. Liu, M. Lv, and Y . Cai. Comatrack: Competitive multi-agent game-theoretic tracking with vision-language-action models.arXiv preprint arXiv:2603.22846, 2026

    arXiv 2026

  11. [11]

    Zhang, K

    J. Zhang, K. Wang, S. Wang, M. Li, H. Liu, S. Wei, Z. Wang, Z. Zhang, and H. Wang. Uni- navid: A video-based vision-language-action model for unifying embodied navigation tasks. InRobotics: Science and Systems, 2025

    2025

  12. [12]

    S. Wang, Y . Luo, X. Chen, A. Luo, D. Li, C. Liu, S. Chen, Y . Zhang, and J. Yu. Vlingnav: Embodied navigation with adaptive reasoning and visual-assisted linguistic memory.arXiv pre-print, 2026. URLhttps://arxiv.org/abs/2601.08665

    arXiv 2026

  13. [13]

    Z. Chu, S. Xie, X. Wu, Y . Shen, M. Luo, Z. Wang, F. Liu, X. Leng, J. Hu, M. Yin, et al. Abot- n0: Technical report on the vla foundation model for versatile embodied navigation.arXiv preprint arXiv:2602.11598, 2026

    arXiv 2026

  14. [14]

    Krantz, E

    J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee. Beyond the nav-graph: Vision- and-language navigation in continuous environments. InEuropean Conference on Computer Vision, pages 104–120. Springer, 2020

    2020

  15. [15]

    Yokoyama, R

    N. Yokoyama, R. Ramrakhya, A. Das, D. Batra, and S. Ha. Hm3d-ovon: A dataset and bench- mark for open-vocabulary object goal navigation. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5543–5550. IEEE, 2024. 9

    2024

  16. [16]

    Z. Chen, Y . Guo, Z. Chu, M. Luo, Y . Shen, M. Sun, J. Hu, S. Xie, K. Yang, P. Shi, et al. So- cialnav: Training human-inspired foundation model for socially-aware embodied navigation. arXiv preprint arXiv:2511.21135, 2025

    arXiv 2025

  17. [17]

    Jiang, L

    Q. Jiang, L. Wu, Z. Zeng, T. Ren, Y . Xiong, Y . Chen, L. Qin, and L. Zhang. Referring to any person. InProc. IEEE Int. Conf. Comput. Vis., pages 21667–21678, 2025

    2025

  18. [18]

    J. Zuo, J. Hong, F. Zhang, C. Yu, H. Zhou, C. Gao, N. Sang, and J. Wang. Plip: Language- image pre-training for person representation learning.Advances in neural information pro- cessing systems, 37:45666–45702, 2024

    2024

  19. [19]

    D. Shah, B. Osinski, B. Ichter, and S. Levine. Lm-nav: Robotic navigation with large pre- trained models of language, vision, and action.arXiv preprint arXiv:2207.04429, 2022

    Pith/arXiv arXiv 2022

  20. [20]

    G. Zhou, Y . Hong, and Q. Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models.arXiv preprint arXiv:2305.16986, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Zhang, K

    J. Zhang, K. Wang, R. Xu, G. Zhou, Y . Hong, X. Fang, Q. Wu, Z. Zhang, and H. Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation.Robotics: Science and Systems, 2024

    2024

  22. [22]

    Cheng, Y

    A.-C. Cheng, Y . Ji, Z. Yang, X. Zou, J. Kautz, E. Biyik, H. Yin, S. Liu, and X. Wang. Navila: Legged robot vision-language-action model for navigation. InRobotics: Science and Systems, 2025

    2025

  23. [23]

    M. Wei, C. Wan, X. Yu, T. Wang, Y . Yang, X. Mao, C. Zhu, W. Cai, H. Wang, Y . Chen, et al. Streamvln: Streaming vision-and-language navigation via slowfast context modeling.arXiv preprint arXiv:2507.05240, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Zhong, P

    F. Zhong, P. Sun, W. Luo, T. Yan, and Y . Wang. Towards distraction-robust active visual tracking. InInternational Conference on Machine Learning, pages 12782–12792. PMLR, 2021

    2021

  25. [25]

    D. Peng, J. Cao, Q. Zhang, and J. Ma. Lovon: Legged open-vocabulary object navigator.arXiv preprint arXiv:2507.06747, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Kazemzadeh, V

    S. Kazemzadeh, V . Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014

    2014

  27. [27]

    H. Qiu, H. Li, T. Zhao, L. Wang, Q. Wu, and F. Meng. Refcrowd: Grounding the target in crowd with referring expressions. InProceedings of the 30th ACM International Conference on Multimedia, pages 4435–4444, 2022

    2022

  28. [28]

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-mark prompting unleashes extraor- dinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Pith/arXiv arXiv 2023

  29. [29]

    K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    H. You, H. Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao, S.-F. Chang, and Y . Yang. Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    C. Ma, Y . Jiang, J. Wu, Z. Yuan, and X. Qi. Groma: Localized visual tokenization for ground- ing multimodal large language models.arXiv preprint arXiv:2404.13013, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Jiang, Y

    Q. Jiang, Y . Yang, Y . Xiong, Y . Chen, Z. Zeng, T. Ren, L. Zhang, et al. Chatrex: Taming multimodal llm for joint perception and understanding.arXiv preprint arXiv:2411.18363, 2024. 10

    Pith/arXiv arXiv 2024

  33. [33]

    S. Yang, T. Qu, X. Lai, Z. Tian, B. Peng, S. Liu, and J. Jia. Lisa++: An improved baseline for reasoning segmentation with large language model.arXiv preprint arXiv:2312.17240, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    Martin-Martin, M

    R. Martin-Martin, M. Patel, H. Rezatofighi, A. Shenoi, J. Gwak, E. Frankel, A. Sadeghian, and S. Savarese. Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments.IEEE transactions on pattern analysis and machine intelligence, 2021

    2021

  35. [35]

    H. Ye, Y . Zhan, W. Situ, G. Chen, J. Yu, Z. Zhao, K. Cai, A. Ajoudani, and H. Zhang. Tpt- bench: A large-scale, long-term and robot-egocentric dataset for benchmarking target person tracking.The International Journal of Robotics Research, page 02783649261447308, 2025

    2025

  36. [36]

    Geiger, P

    A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. InConference on Computer Vision and Pattern Recognition (CVPR), 2012

    2012

  37. [37]

    J. W. Bae, J. Kim, J. Yun, C. Kang, J. Choi, C. Kim, J. Lee, J. Choi, and J. W. Choi. Sit dataset: socially interactive pedestrian trajectory dataset for social navigation robots.Advances in neural information processing systems, 36:24552–24563, 2023

    2023

  38. [38]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  39. [39]

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. InProc. IEEE Int. Conf. Comput. Vis., pages 11975–11986, 2023

    2023

  40. [40]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    Jocher and J

    G. Jocher and J. Qiu. Ultralytics yolo11.https://github.com/ultralytics/ ultralytics, 2024

    2024

  42. [42]

    X. Puig, E. Undersander, A. Szot, M. D. Cote, T.-Y . Yang, R. Partsey, R. Desai, A. W. Clegg, M. Hlavac, S. Y . Min, et al. Habitat 3.0: A co-habitat for humans, avatars and robots.arXiv preprint arXiv:2310.13724, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    K. Li, Y . He, Y . Wang, Y . Li, W. Wang, P. Luo, Y . Wang, L. Wang, and Y . Qiao. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22216–22227, 2024

    2024

  44. [44]

    Introducing 4o image generation.https://openai.com/index/ introducing-4o-image, 2024

    OpenAI. Introducing 4o image generation.https://openai.com/index/ introducing-4o-image, 2024. Accessed: 2025-04-29

    2024

  45. [45]

    Gupta, S

    M. Gupta, S. Kumar, L. Behera, and V . K. Subramanian. A novel vision-based tracking algo- rithm for a human-following mobile robot.IEEE Transactions on Systems, Man, and Cyber- netics: Systems, 47(7):1415–1427, 2016

    2016

  46. [46]

    K.-H. Zeng, Z. Zhang, K. Ehsani, R. Hendrix, J. Salvador, A. Herrasti, R. Girshick, A. Kem- bhavi, and L. Weihs. Poliformer: Scaling on-policy rl with transformers results in masterful navigators. In8th Annual Conference on Robot Learning, 2024

    2024

  47. [47]

    Zhang, G

    J. Zhang, G. Zhou, H. Yin, Y . Huang, Z. Lei, Q. Peng, H. Yuan, J. Zhang, X. Guo, X. Chen, et al. Qwen-robotnav technical report: A scalable navigation model designed for an agentic navigation system.arXiv preprint arXiv:2606.18112, 2026

    Pith/arXiv arXiv 2026

  48. [48]

    K. Lee, H. Ying, and T. Zhao. Opentrackvla: Open-source visual language action model for visual navigation and following.https://github.com/om-ai-lab/OpenTrackVLA, 2025

    2025

  49. [49]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 11

    Pith/arXiv arXiv 2025

  50. [50]

    Zhang, P

    Y . Zhang, P. Sun, Y . Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang. Bytetrack: Multi-object tracking by associating every detection box. InEuropean conference on computer vision, pages 1–21. Springer, 2022

    2022

  51. [51]

    H. Ye, J. Zhao, Y . Zhan, W. Chen, L. He, and H. Zhang. Person re-identification for robot person following with online continual learning.IEEE Robotics and Automation Letters, 2024

    2024

  52. [52]

    Track the person wearing a red and yellow superhero suit

    H. Ye, J. Zhao, Y . Pan, W. Cherr, L. He, and H. Zhang. Robot person following under partial occlusion. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7591–7597. IEEE, 2023. A Training Details A.1 Expert Tracking Data Curation We source expert tracking trajectories from the EVT-Bench training split [1] in Habitat 3.0 [42]. B...

    2023