Pith. sign in

REVIEW 3 major objections 55 references

Training BEV instance predictors with vision-language semantic maps makes future vehicle forecasts more accurate without raising inference cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 13:08 UTC pith:FSFDVJZJ

load-bearing objection Clean training-only recipe that adds a real but tiny boost to BEV future-instance prediction via offline CLIP maps; the 'fundamental' and SOTA claims overshoot the +0.4 IoU evidence. the 3 major comments →

arxiv 2607.04812 v1 pith:FSFDVJZJ submitted 2026-07-06 cs.CV

TGRIP: A Text-Guided Approach to Vehicle Instance Prediction in Autonomous Driving

Miguel Antunes-Garc\'ia , Santiago Montiel-Mar\'in , Fabio S\'anchez-Garc\'ia , Rodrigo Guti\'errez-Moreno , Rafael Barea , Luis M. Bergasa This is my paper
classification cs.CV
keywords bird's-eye viewinstance predictionvision-language modelsautonomous drivingsemantic supervisionCLIPend-to-end motion forecastingnuScenes
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

End-to-end bird’s-eye-view instance prediction for autonomous driving usually trains only on geometry—occupancy and flow—so every agent is treated as a generic moving blob. The paper argues this leaves the model unable to resolve ambiguities that depend on what an object is, such as overtaking or intersection behavior. TGRIP builds dense semantic BEV maps offline by cropping each vehicle with ground-truth 3D boxes, embedding the crop with a vision-language model, and painting that embedding into the BEV grid. An auxiliary head is trained to match those maps, then discarded, so the shared backbone keeps the semantic structure while the deployed model stays the same size and speed. On nuScenes the approach beats purely geometric baselines, supporting the claim that semantic enrichment is a fundamental ingredient for robust motion prediction.

Core claim

Dense, instance-level semantic BEV maps distilled from vision-language models, used solely as auxiliary training supervision, produce shared BEV features that are both geometrically consistent and semantically discriminative. After the semantic head is removed at test time, those features yield higher IoU and video panoptic quality for future vehicle instance prediction on nuScenes than geometric-only training, establishing a new long-range state of the art without any inference overhead.

What carries the argument

The teacher–student semantic BEV pipeline: per-instance CLIP or SigLIP embeddings are extracted from multi-camera crops of ground-truth 3D boxes, rasterized into dense BEV ground-truth maps, and used to supervise an auxiliary residual head via cosine similarity so the shared backbone learns semantically aligned features.

Load-bearing premise

The lasting gain assumes that forcing the shared backbone to match offline CLIP embeddings of ground-truth crops continues to help flow and segmentation after the semantic head is thrown away at test time.

What would settle it

Rerun the identical two-stage training but replace the CLIP embeddings with random vectors of the same dimension (or pure noise); if the IoU and VPQ gains over the geometric baseline vanish or reverse, the claim that semantic content is doing the work is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Instance association and long-range forecasting improve in cluttered or nighttime scenes once the backbone encodes object identity, not only occupancy.
  • Any BEV prediction pipeline can adopt offline VLM teachers without paying extra latency or parameters at deployment.
  • Fine-grained visual embeddings outperform static class-text embeddings as the supervision signal.
  • Even lightweight CLIP variants already capture enough semantic structure for the gains, so teacher cost can stay modest.
  • The resulting BEV features become linearly separable by vehicle category and respond to free-form text queries such as “truck” or “bus”.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same cosine-alignment recipe works when teacher crops are produced by open-vocabulary detectors instead of dataset boxes, the method could scale to unlabeled driving video.
  • Semantic BEV tokens shaped this way are natural structured inputs for downstream language-conditioned planners that currently receive only geometry or hand-crafted descriptors.
  • The modest absolute lifts suggest the main value appears where pure geometry is ambiguous; ablations on deliberately occluded or multi-modal scenes would quantify that claim.
  • Knowledge distillation of the already-enriched backbone into a smaller student could move the benefit onto embedded automotive hardware without re-running the teacher.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. TGRIP proposes a teacher–student framework for end-to-end BEV vehicle instance prediction that injects dense semantic supervision derived from frozen vision–language models (CLIP / SigLIP2). Offline, 3D GT boxes are used to crop multi-camera images, extract per-instance embeddings, and rasterize them into semantic BEV maps (Algorithm 1 / §III-B). An auxiliary residual head is trained with cosine-similarity loss on the present frame only (§III-C,D, Eq. 1); the head is discarded at inference so the deployed model matches the geometric baseline (BEVPredFormer + EfficientViT + BEVFormer). On nuScenes validation the method reports long-range IoU 41.3 / VPQ 34.3 versus the authors’ geometric baseline 40.9 / 33.3 (Table I), with multi-seed confirmation and ablations over teacher architecture and visual vs. text cues (Tables II–III). The central claim is that this semantic enrichment of the shared backbone is fundamental for robust future instance prediction.

Significance. If the reported gains truly reflect a transferable inductive bias rather than privileged teacher access, the work supplies a practical, zero-inference-cost recipe for enriching dense BEV motion predictors with open-vocabulary priors—an underexplored direction relative to static open-vocabulary occupancy. Strengths include a clean two-stage training protocol, public code, multi-seed statistics, and systematic ablations of teacher size and cue type. The absolute deltas remain small, so the result is incremental rather than transformative, but it is a concrete, reproducible step toward semantically aware end-to-end prediction.

major comments (3)
  1. §III-B / Algorithm 1 and Table I: the teacher is given perfect 3D GT boxes to produce instance-aligned crops and BEV rasterization that exactly match the geometric GT used for flow/segmentation. The student never sees these privileged localizations at inference. The measured +0.4 IoU / +1.0 VPQ (and multi-seed confirmation) may therefore arise from geometric-semantic co-supervision rather than a lasting, geometry-only inductive bias. A control that replaces CLIP embeddings by pure geometric or random vectors of identical spatial support is required to isolate the semantic contribution; without it the claim that “semantic enrichment is fundamental” is under-supported.
  2. Table I and the SOTA claim: absolute gains over the authors’ own baseline are modest (+0.4 IoU / +1.0 VPQ long-range). DMP still leads short-range VPQ (57.5 vs 56.3) and several prior methods lack official code, so direct comparison is incomplete. The language “surpasses existing state-of-the-art models” and “fundamental element” should be tempered to the observed effect size and the controlled baseline comparison actually performed.
  3. §III-C,D: cosine supervision is applied only to the present BEV frame, yet the claimed benefit is improved future flow and instance association (Tout = 6). No ablation isolates whether the semantic signal improves future-frame metrics more than present-frame segmentation alone. A temporal ablation (semantic loss on present only vs. present+future) would clarify whether the inductive bias actually propagates into the motion heads.

Circularity Check

0 steps flagged

Empirical systems paper with external nuScenes metrics; minor self-baseline only, no derivation-by-construction circularity.

full rationale

TGRIP is a standard teacher-student empirical architecture paper, not a first-principles derivation. The load-bearing claims (IoU/VPQ gains of +0.4/+1.0 long-range over the geometric baseline, new SOTA vs external methods) are measured on the held-out nuScenes validation split using the official external metrics defined in Eqs. 2-3; those metrics are independent of the training losses (Smooth-L1 flow, CE segmentation, L2 centerness, cosine L_sem). The semantic teacher (Algorithm 1 / §III-B) is constructed offline from dataset-provided 3D GT boxes + a frozen public CLIP/SigLIP encoder and is discarded at inference; the student is never scored on the teacher embeddings themselves. Ablations (Tables II-III) vary the teacher architecture and cue type while keeping the evaluation fixed. The only self-reference is the geometric baseline BEVPredFormer (authors' concurrent arXiv), which is a normal comparison and does not force the reported deltas by construction. No self-definitional loop, no fitted parameter renamed as prediction, no uniqueness theorem imported from prior self-work, and no ansatz smuggled via citation. Score 1 solely for the minor self-baseline; central experimental content is independent and externally falsifiable.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The central claim is empirical and rests on standard AD/CV assumptions plus a few training knobs. No new physical entities are postulated. Load-bearing content is: (1) frozen VLM embeddings of GT-cropped instances are a useful teacher for motion features; (2) multi-task geometric losses plus cosine semantic loss improve shared BEV features; (3) evaluation protocol (nuScenes vehicles, fixed ranges, IoU/VPQ). Free parameters are ordinary optimizer/architecture choices; none are fitted to invent the metric itself.

free parameters (6)
  • stage-1 max learning rate
    Chosen as 3e-4 with One-Cycle AdamW for 70 epochs; not derived, affects convergence of the geometric baseline that later gains are measured against.
  • stage-2 max learning rate
    Reduced to 3e-5 for 40-epoch semantic fine-tuning; hand-chosen to 'merge' semantic knowledge without destroying stage-1 weights.
  • multi-task loss weights λ1..λ4 (dynamic)
    Dynamic weighting balances flow, segmentation, centerness, and semantic cosine losses (Eq. 1); schedule details are underspecified and can change which head dominates.
  • temporal discount γt for segmentation loss
    Future frames are down-weighted with a decreasing γt∈(0,1]; exact schedule is a free design choice that affects long-horizon metrics.
  • semantic residual depth M and CCLIP
    M=2 residual blocks and CCLIP∈{512,768} set capacity of the auxiliary head; selected by authors, not predicted by theory.
  • BEV grid / range (200×200 at 0.5 m or 0.15 m)
    Spatial resolution and 50 m vs 15 m ranges are protocol choices that define the reported IoU/VPQ numbers.
axioms (5)
  • domain assumption Frozen CLIP/SigLIP image embeddings of GT-cropped vehicles encode instance-specific cues (appearance, type, orientation) that are useful for future BEV instance association and motion.
    Core hypothesis of §I and §III-B/C; without it, cosine supervision is noise.
  • domain assumption Dense BEV occupancy + flow is a sufficient output interface for end-to-end instance prediction (post-process flow to propagate IDs).
    Inherited from FIERY/PowerBEV/BEVPredFormer line (§II-C, §III-A); metrics IoU/VPQ assume this representation.
  • ad hoc to paper Improvements measured on the shared backbone after removing the semantic head at inference reflect true representation enrichment, not train-time co-adaptation that requires the head.
    Stated design goal in §III-C and conclusion; not independently proven beyond final metric deltas.
  • domain assumption nuScenes vehicle supercategory labels and official train/val split are an adequate proxy for 'robust motion prediction' claims.
    Entire §IV evaluation; no other datasets or agent classes.
  • standard math Standard multi-task losses (Smooth-L1 flow, weighted CE segmentation, L2 centerness, cosine semantic) are valid optimization targets for the claimed metrics.
    §III-D; conventional CV losses, not re-derived.
invented entities (2)
  • Semantic-enhanced BEV ground-truth maps V (CLIP embeddings rasterized into occupied BEV cells) no independent evidence
    purpose: Provide dense open-vocabulary-style supervision aligned to the instance prediction grid without requiring new human labels beyond existing 3D boxes.
    Defined by Algorithm 1 / Fig. 3; a methodological construct, not an external physical object. No independent measurement outside this training pipeline.
  • TGRIP auxiliary semantic branch (temporal collapse + residual Conv blocks → CCLIP map) no independent evidence
    purpose: Distill teacher embeddings into shared FBEV features during training only.
    §III-C architectural invention; discarded at inference, so its only evidence is the reported metric lift.

pith-pipeline@v1.1.0-grok45 · 20871 in / 3857 out tokens · 39286 ms · 2026-07-11T13:08:18.982496+00:00 · methodology

0 comments
read the original abstract

Bird's-Eye View (BEV) end-to-end instance prediction has emerged as a robust paradigm for autonomous driving perception, effectively mitigating the error propagation inherent in traditional modular pipelines. However, current state-of-the-art approaches rely predominantly on geometric supervision, such as occupancy regression and optical flow, effectively treating scene agents as generic moving obstacles. This absence of explicit semantic awareness imposes limitations on the capacity of the model to solve ambiguities in complex scenarios, particularly those where object-specific behavior is essential for accurate forecasting (e.g. overtaking, intersections). In this paper, we introduce Text-Guided Representation for Instance Prediction (TGRIP), a novel framework that bridges this gap by injecting rich semantic priors into the instance prediction loop. The proposed teacher-student pipeline employs Vision-Language Foundation Models to generate dense, semantic-enhanced BEV maps from multi-camera images. These maps serve as auxiliary supervision during training, guiding the network to learn spatio-temporal representations that are not only geometrically consistent but also semantically discriminative. To the best of our knowledge, this represents the first attempt to unify semantic guidance with the temporal task of future instance prediction. The experimental results demonstrate that TGRIP surpasses existing state-of-the-art models in nuScenes, validating the hypothesis that semantic enrichment is a fundamental element for robust, end-to-end motion prediction. Code is available on https://github.com/miguelag99/TGRIP.

Figures

Figures reproduced from arXiv: 2607.04812 by Fabio S\'anchez-Garc\'ia, Luis M. Bergasa, Miguel Antunes-Garc\'ia, Rafael Barea, Rodrigo Guti\'errez-Moreno, Santiago Montiel-Mar\'in.

Figure 1
Figure 1. Figure 1: TGRIP incorporates supplementary semantic supervision only during the training process by leveraging a foundational Vision-Language model to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the TGRIP framework. A specialized pipeline is employed to generate semantic BEV ground truth, which is then used to supervise an [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the BEV semantic ground truth generation pipeline. Object-level crops are first extracted from the input images using the 3D ground [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Instance prediction qualitative results in nuScenes of TGRIP and the baseline without semantic supervision. The present frame ego vehicle is [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative visualization of TGRIP semantic maps on nuScenes. The [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 15 linked inside Pith

  1. [1]

    Intentnet: Learning to predict intention from raw sensor data,

    S. Casas, W. Luo, and R. Urtasun, “Intentnet: Learning to predict intention from raw sensor data,” inConference on robot learning. PMLR, 2018, pp. 947–956

    2018

  2. [2]

    Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction,

    Y . Chai, B. Sapp, M. Bansal, and D. Anguelov, “Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction,” arXiv preprint arXiv:1910.05449, 2019

    Pith/arXiv arXiv 1910

  3. [3]

    Efficient baselines for motion prediction in autonomous driving,

    C. Gómez-Huélamo, M. V . Conde, R. Barea, M. Ocaña, and L. M. Bergasa, “Efficient baselines for motion prediction in autonomous driving,”IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 5, pp. 4192–4205, 2023

    2023

  4. [4]

    FIERY: Future instance segmentation in bird’s-eye view from surround monocular cameras,

    A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, V . Badrinarayanan, R. Cipolla, and A. Kendall, “FIERY: Future instance segmentation in bird’s-eye view from surround monocular cameras,” inProceedings of the International Conference on Computer Vision (ICCV), 2021

    2021

  5. [5]

    Powerbev: A powerful yet lightweight framework for instance prediction in bird’s-eye view,

    P. Li, S. Ding, X. Chen, N. Hanselmann, M. Cordts, and J. Gall, “Powerbev: A powerful yet lightweight framework for instance prediction in bird’s-eye view,” inProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, E. Elkind, Ed. International Joint Conferences on Artificial Intelligence Organization, 8 2023, ...

    doi:10.24963/ijcai.2023/120 2023

  6. [6]

    Fast and efficient transformer- based method for bird’s eye view instance prediction,

    M. Antunes-García, L. M. Bergasa, S. Montiel-Marín, R. Barea, F. Sánchez-García, and A. Llamazares, “Fast and efficient transformer- based method for bird’s eye view instance prediction,” in2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC), 2024, pp. 1269–1274

    2024

  7. [7]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  8. [8]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Dollar, and R. Girshick, “Segment anything,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp. 4015–4026

    2023

  9. [9]

    Pop-3d: Open-vocabulary 3d occupancy prediction from images,

    A. V obecky, O. Siméoni, D. Hurych, S. Gidaris, A. Bursuc, P. Pérez, and J. Sivic, “Pop-3d: Open-vocabulary 3d occupancy prediction from images,”Advances in Neural Information Processing Systems, vol. 36, pp. 50 545–50 557, 2023

    2023

  10. [10]

    nuscenes: A multimodal dataset for autonomous driving,

    H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Kr- ishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,”arXiv preprint arXiv:1903.11027, 2019

    Pith/arXiv arXiv 1903

  11. [11]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” inInternational conference on machine learning. PMLR, 2021, pp. 4904–4916

    2021

  12. [12]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 11 975–11 986

    2023

  13. [13]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features,

    M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdul- mohsin, N. Parthasarathy, T. Evans, L. Beyer, Y . Xia, B. Mustafa, et al., “Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features,”arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Maskclip: Masked self-distillation advances contrastive language-image pretraining,

    X. Dong, J. Bao, Y . Zheng, T. Zhang, D. Chen, H. Yang, M. Zeng, W. Zhang, L. Yuan, D. Chen,et al., “Maskclip: Masked self-distillation advances contrastive language-image pretraining,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 10 995–11 005

    2023

  15. [15]

    Zegclip: Towards adapting clip for zero-shot semantic segmentation,

    Z. Zhou, Y . Lei, B. Zhang, L. Liu, and Y . Liu, “Zegclip: Towards adapting clip for zero-shot semantic segmentation,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 11 175–11 185

    2023

  16. [16]

    Emerging properties in self-supervised vision transformers,

    M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9650–9660

    2021

  17. [17]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby,et al., “Dinov2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Siméoni, H

    O. Siméoni, H. V . V o, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V . Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, F. Massa, D. Haziza, L. Wehrstedt, J. Wang, T. Darcet, T. Moutakanni, L. Sentana, C. Roberts, A. Vedaldi, J. Tolan, J. Brandt, C. Couprie, J. Mairal, H. Jégou, P. Labatut, and P. Bojanowski, “Dinov3,” 2025. [Online]. Available: https...

    Pith/arXiv arXiv 2025

  19. [19]

    Sam 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson,et al., “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408.00714, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Sam 3: Segment anything with concepts,

    N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V . Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. Rädle, T. Afouras, E. Mavroudi, K. Xu, T.-H. Wu, Y . Zhou, L. Momeni, R. Hazra, S. Ding, S. Vaze, F. Porcher, F. Li, S. Li, A. Kamath, H. K. Cheng, P. Dollár, N. Ravi, K. Sae...

    Pith/arXiv arXiv 2025

  21. [21]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, pp. 34 892– 34 916, 2023

    2023

  22. [22]

    Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking,

    M. Li, Y . Zhang, D. Long, K. Chen, S. Song, S. Bai, Z. Yang, P. Xie, A. Yang, D. Liu,et al., “Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking,”arXiv preprint arXiv:2601.04720, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling,

    Z. Chen, W. Wang, Y . Cao, Y . Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu,et al., “Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling,” arXiv preprint arXiv:2412.05271, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,

    J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y . Duan, W. Su, J. Shao,et al., “Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,”arXiv preprint arXiv:2504.10479, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Openworld- sam: Extending sam2 for universal image segmentation with language prompts,

    S. Xiao, R. Kabra, Y . Li, D. Lee, J. Carreira, and P. Panda, “Openworld- sam: Extending sam2 for universal image segmentation with language prompts,”arXiv preprint arXiv:2507.05427, 2025

    arXiv 2025

  26. [26]

    Lposs: Label propagation over patches and pixels for open-vocabulary semantic segmentation,

    V . Stojni´c, Y . Kalantidis, J. Matas, and G. Tolias, “Lposs: Label propagation over patches and pixels for open-vocabulary semantic segmentation,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 9794–9803

    2025

  27. [27]

    Tri-perspective view for vision-based 3d semantic occupancy prediction,

    Y . Huang, W. Zheng, Y . Zhang, J. Zhou, and J. Lu, “Tri-perspective view for vision-based 3d semantic occupancy prediction,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 9223–9232

    2023

  28. [28]

    Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction,

    Y . Zhang, Z. Zhu, and D. Du, “Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 9433–9443

    2023

  29. [29]

    Ovo: Open- vocabulary occupancy,

    Z. Tan, Z. Dong, C. Zhang, W. Zhang, H. Ji, and H. Li, “Ovo: Open- vocabulary occupancy,”arXiv preprint arXiv:2305.16133, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Talk2bev: Language-enhanced bird’s-eye view maps for autonomous driving,

    T. Choudhary, V . Dewangan, S. Chandhok, S. Priyadarshan, A. Jain, A. K. Singh, S. Srivastava, K. M. Jatavallabhula, and K. M. Krishna, “Talk2bev: Language-enhanced bird’s-eye view maps for autonomous driving,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 16 345–16 352

    2024

  31. [31]

    Bevformer: learning bird’s-eye-view representation from lidar- camera via spatiotemporal transformers,

    Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, and J. Dai, “Bevformer: learning bird’s-eye-view representation from lidar- camera via spatiotemporal transformers,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  32. [32]

    Bevsegformer: Bird’s eye view semantic segmentation from arbitrary camera rigs,

    L. Peng, Z. Chen, Z. Fu, P. Liang, and E. Cheng, “Bevsegformer: Bird’s eye view semantic segmentation from arbitrary camera rigs,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 5935–5943

    2023

  33. [33]

    Gaussiancar: Gaussian splatting for efficient camera-radar fusion,

    S. Montiel-Marín, M. Antunes-García, F. Sánchez-García, A. Llamazares, H. Caesar, and L. M. Bergasa, “Gaussiancar: Gaussian splatting for efficient camera-radar fusion,” 2026. [Online]. Available: https://arxiv.org/abs/2602.08784

    arXiv 2026

  34. [34]

    Pointbev: A sparse approach for bev predictions,

    L. Chambon, E. Zablocki, M. Chen, F. Bartoccioni, P. Pérez, and M. Cord, “Pointbev: A sparse approach for bev predictions,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 15 195–15 204

    2024

  35. [35]

    Fb- bev: Bev representation from forward-backward view transformations,

    Z. Li, Z. Yu, W. Wang, A. Anandkumar, T. Lu, and J. M. Alvarez, “Fb- bev: Bev representation from forward-backward view transformations,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 6919–6928

    2023

  36. [36]

    Is ego status all you need for open-loop end-to-end autonomous driving?

    Z. Li, Z. Yu, S. Lan, J. Li, J. Kautz, T. Lu, and J. M. Alvarez, “Is ego status all you need for open-loop end-to-end autonomous driving?” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 864–14 873

    2024

  37. [37]

    Lmdrive: Closed-loop end-to-end driving with large language models,

    H. Shao, Y . Hu, L. Wang, G. Song, S. L. Waslander, Y . Liu, and H. Li, “Lmdrive: Closed-loop end-to-end driving with large language models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 120–15 130

    2024

  38. [38]

    Bevdriver: Leveraging bev maps in llms for robust closed-loop driving,

    K. Winter, M. Azer, and F. B. Flohr, “Bevdriver: Leveraging bev maps in llms for robust closed-loop driving,”arXiv preprint arXiv:2503.03074, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Safety-enhanced autonomous driving using interpretable sensor fusion transformer,

    H. Shao, L. Wang, R. Chen, H. Li, and Y . Liu, “Safety-enhanced autonomous driving using interpretable sensor fusion transformer,” in Conference on Robot Learning. PMLR, 2023, pp. 726–737

    2023

  40. [40]

    Reasonnet: End-to-end driving with temporal and global reasoning,

    H. Shao, L. Wang, R. Chen, S. L. Waslander, H. Li, and Y . Liu, “Reasonnet: End-to-end driving with temporal and global reasoning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 13 723–13 733

    2023

  41. [41]

    Vectornet: Encoding hd maps and agent dynamics from vectorized representation,

    J. Gao, C. Sun, H. Zhao, Y . Shen, D. Anguelov, C. Li, and C. Schmid, “Vectornet: Encoding hd maps and agent dynamics from vectorized representation,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11 525–11 533

    2020

  42. [42]

    Learning lane graph representations for motion forecasting,

    M. Liang, B. Yang, R. Hu, Y . Chen, R. Liao, S. Feng, and R. Urtasun, “Learning lane graph representations for motion forecasting,” in European Conference on Computer Vision. Springer, 2020, pp. 541– 556

    2020

  43. [43]

    Scene transformer: A unified architecture for predicting multiple agent trajectories,

    J. Ngiam, B. Caine, V . Vasudevan, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal,et al., “Scene transformer: A unified architecture for predicting multiple agent trajectories,”arXiv preprint arXiv:2106.08417, 2021

    Pith/arXiv arXiv 2021

  44. [44]

    Hivt: Hierarchical vector transformer for multi-agent motion prediction,

    Z. Zhou, L. Ye, J. Wang, K. Wu, and K. Lu, “Hivt: Hierarchical vector transformer for multi-agent motion prediction,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 8823–8833

    2022

  45. [45]

    Motion transformer with global intention localization and local movement refinement,

    S. Shi, L. Jiang, D. Dai, and B. Schiele, “Motion transformer with global intention localization and local movement refinement,”Advances in Neural Information Processing Systems, vol. 35, pp. 6531–6543, 2022

    2022

  46. [46]

    Query-centric trajectory prediction,

    Z. Zhou, J. Wang, Y .-H. Li, and Y .-K. Huang, “Query-centric trajectory prediction,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 17 863–17 873

    2023

  47. [47]

    Vip3d: End-to-end visual trajectory prediction via 3d agent queries,

    J. Gu, C. Hu, T. Zhang, X. Chen, Y . Wang, Y . Wang, and H. Zhao, “Vip3d: End-to-end visual trajectory prediction via 3d agent queries,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5496–5506

    2023

  48. [48]

    Detra: A unified model for object detection and trajectory forecasting,

    S. Casas, B. Agro, J. Mao, T. Gilles, A. Cui, T. Li, and R. Urtasun, “Detra: A unified model for object detection and trajectory forecasting,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 326–342

    2024

  49. [49]

    Beverse: Unified perception and prediction in birds-eye-view for vision- centric autonomous driving,

    Y . Zhang, Z. Zhu, W. Zheng, J. Huang, G. Huang, J. Zhou, and J. Lu, “Beverse: Unified perception and prediction in birds-eye-view for vision- centric autonomous driving,”arXiv preprint arXiv:2205.09743, 2022

    Pith/arXiv arXiv 2022

  50. [50]

    Stretchbev: Stretching future instance prediction spatially and temporally,

    A. K. Akan and F. Güney, “Stretchbev: Stretching future instance prediction spatially and temporally,”European Conference on Computer Vision (ECCV), 2022

    2022

  51. [51]

    St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning,

    S. Hu, L. Chen, P. Wu, H. Li, J. Yan, and D. Tao, “St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning,” inEuropean Conference on Computer Vision (ECCV), 2022

    2022

  52. [52]

    Dmp: Difference-guided motion prediction for vision-centric autonomous driving,

    Y . Chen, C. Lin, X. Duan, J. Zhou, K. Guo, D. Zhao, D. Cao, and D. Tian, “Dmp: Difference-guided motion prediction for vision-centric autonomous driving,”IEEE Transactions on Intelligent Transportation Systems, vol. 26, no. 6, pp. 9094–9108, 2025

    2025

  53. [53]

    Bevpredformer: Spatio-temporal attention for bev instance prediction in autonomous driving,

    M. Antunes-García, S. Montiel-Marín, F. Sánchez-García, R. Gutiérrez- Moreno, R. Barea, and L. M. Bergasa, “Bevpredformer: Spatio-temporal attention for bev instance prediction in autonomous driving,” 2026. [Online]. Available: https://arxiv.org/abs/2604.02930

    Pith/arXiv arXiv 2026

  54. [54]

    Efficientvit: Memory efficient vision transformer with cascaded group attention,

    X. Liu, H. Peng, N. Zheng, Y . Yang, H. Hu, and Y . Yuan, “Efficientvit: Memory efficient vision transformer with cascaded group attention,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 14 420–14 430

    2023

  55. [55]

    St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning,

    S. Hu, L. Chen, P. Wu, H. Li, J. Yan, and D. Tao, “St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 533–549

    2022