Pith. sign in

REVIEW 5 major objections 6 minor 43 references

Temporal Object Captioning for Street Scene Videos from LiDAR Tracks

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training a video captioning model on template captions generated automatically from LiDAR tracks improves temporal understanding and reduces visual bias across three driving-scene datasets.

desk verdict Useful automated LiDAR-to-caption pipeline, but the central claim about reduced visual bias rests on an unvalidated, self-referential metric. read the letter →

arxiv 2505.16594 v1 pith:6EPXRTO3 submitted 2025-05-22 cs.CV cs.LG

Vignesh Gopinathan , Urs Zimmermann , Michael Arnold , Matthias Rottmann This is my paper
classification cs.CVcs.LG
keywords videocaptioningLiDARtemporalunderstandingvisualbiasstreetscenestemplatecaptionsMeasureautonomousdriving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that video captioning models can learn genuine temporal semantics when supervised with template captions generated automatically from LiDAR object tracks. The authors build a rule-based pipeline that turns tracks into sentences describing lane position and relative motion, such as approaching, moving away, or changing lanes, and then train SwinBERT on front-camera frames using those captions. Across a proprietary dataset, Waymo, and NuScenes, the trained model reports higher BLEU4, CIDEr, and SPICE scores than InternVideo, ViCLIP, and CLIP, and it shows the smallest drop in retrieval quality under the paper's new Visual Bias Measure. If correct, the result means a fully automated, human-free labeling procedure can supply scalable temporal supervision for driving-scene video understanding.

What carries the argument

The load-bearing mechanism is the rule-based captioning pipeline paired with the Visual Bias Measure. From LiDAR tracks, each neighbor object receives a concatenated tag combining object class, lane tag, and motion tag at every timestamp; the tag time series is collapsed into a unified sequence of distinct behavioral states, and each state is rendered as a sentence from a template pool with 63 first-sentence options and 78 follow-up options. The Visual Bias Measure, VBM = 100*(B_k - C_k)/B_k, is the percentage drop in retrieval BLEU4 when top-k neighbors recorded within a predefined time window of the query are excluded, with lower VBM taken to indicate reliance on temporal semantics rather than visual similarity. As an auxiliary mechanism, LiDAR-derived object masks are produced by projecting points inside each bounding box onto the image plane and wrapping them with a convex-hull algorithm, which further lowers VBM, especially on Waymo and NuScenes.

What would settle it

Compute the Visual Bias Measure for a model trained on the same front-camera clips but with captions randomly reassigned to different clips, destroying the caption-motion correspondence; if the shuffled model's VBM is as low as the trained model's, the metric is not detecting temporal understanding. A complementary check is to have human raters judge whether retrieved neighbors share the action described and compare those ratings with VBM rankings: low correlation would invalidate the metric.

Watch

Extended reading notes

Core claim

The paper's central claim is that LiDAR-derived template captions, used as supervision for a video captioning model, reduce the visual/static bias in the model's embedding space and improve its temporal understanding. The procedure extracts object tracks from raw LiDAR with an off-the-shelf 3D detector and tracker, assigns each tracked object lane and motion tags by thresholding yaw, lateral position, and relative distance changes, compresses the resulting tag time series into a unified sequence of discrete behavioral states, and maps each state to a sentence from a template pool. Training SwinBERT with masked language modeling on front-camera RGB frames and these captions yields the lowest Visual Bias Measure across the proprietary, Waymo, and NuScenes datasets and the highest captioning scores against InternVideo, ViCLIP, CLIP, and VGG19. The authors interpret this as evidence that the model's embeddings organize by action semantics such as lane, approach, and away rather than by static appearance, supported by UMAP cluster structure and qualitative retrieval examples.

Load-bearing premise

The load-bearing premise is that the Visual Bias Measure, a relative drop in retrieval BLEU4 when nearby-in-time videos are excluded, genuinely measures temporal understanding rather than some other property of the embedding space; a second fragile link is that the off-the-shelf 3D tracker used to build the captions is accurate enough that its errors do not dominate the labels.

Editorial extensions

If this is right

  • Training on LiDAR-derived template captions improves captioning quality and lowers visual bias on datasets with camera setups different from the training data, so the method should transfer to any LiDAR-plus-camera driving dataset.
  • Because LiDAR is used only for label generation and not at inference, the resulting captioning model can run on camera-only systems.
  • Mask-augmented frames built from LiDAR point clouds improve the Visual Bias Measure by roughly 5 percentage points on Waymo and 2 percentage points on NuScenes, providing a cheap additional supervision signal.
  • The template space is combinatorial, yielding thousands of distinct multi-sentence captions from a small set of rules, so the pipeline can generate large pseudo-ground-truth datasets without human annotation.
  • A model trained only on single-sentence captions retrieves videos described by two-sentence captions better than the baselines, indicating zero-shot generalization to more complex temporal descriptions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Visual Bias Measure, if validated against human judgments or downstream tasks, could become a general diagnostic for temporal bias in any video-language model, not just captioning models.
  • The same rule-based captioning pipeline could be applied to other sensor modalities, such as radar or stereo depth, whenever object tracks are available, extending it beyond LiDAR-equipped datasets.
  • A testable extension is to add an LLM paraphrasing stage to the template captions and check whether linguistic diversity improves without lowering temporal grounding or worsening VBM.
  • The claim that template captions are sufficient for action semantics suggests that linguistic variety may matter less than precise, structured supervision for ADAS-oriented video understanding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents an automated LiDAR-based caption generation pipeline for street-scene videos. A 3D detector/tracker extracts object tracks from LiDAR, and a rule-based system assigns lane and motion tags, which are converted into template captions. SwinBERT is trained on front-camera RGB frames supervised by these captions, using only single-sentence captions from a proprietary dataset. The authors evaluate zero-shot caption prediction on proprietary data, Waymo, and NuScenes (Table V), and propose a new Visual Bias Measure (VBM, Eq. 2) to claim reduced visual bias and improved temporal understanding. The paper also studies an automated LiDAR-based object-masking augmentation and provides qualitative embedding analyses.

Significance. The proposed pipeline is fully automated and scalable, and the zero-shot evaluation on two public datasets is a useful methodological choice. If the claims were supported, the work would contribute a practical way to generate temporally grounded driving captions without human annotation. However, the central evidence is not conclusive: the headline comparison in Table V uses different protocols for the proposed model and the baselines, and the VBM metric is unvalidated and confounded by candidate-pool changes. These issues prevent the paper from establishing the claimed reduction in visual bias.

major comments (5)
  1. [Section IV-B, Table V] The headline captioning comparison is not apples-to-apples. The proposed model generates captions, whereas InternVideo, ViCLIP, and CLIP are evaluated by selecting the best match from the caption pool (as the text states: 'the baseline models were not required to generate captions'). BLEU4, CIDEr, and SPICE computed against the template captions therefore mix a generation task with a retrieval task. The reported margins (e.g., '3x in BLEU4') do not establish that the proposed model is a better captioner. Please re-run the baselines under an identical generation protocol, or evaluate all methods by retrieval from an identical candidate set, and report both settings separately.
  2. [Section III-C, Eq. (2), Section IV-C] The Visual Bias Measure is confounded by the retrieval pool change. In the constrained setting, temporally nearby videos are excluded, so C_k is computed over a smaller and likely differently distributed set of template captions; any retrieval method would show some BLEU4 drop under this change. The paper does not report chance-level BLEU4, a pool-size-matched random baseline, or any normalization. Moreover, VBM is never validated against human judgments, established bias metrics, or downstream tasks. Without these controls, the claim that LiDAR-supervised training 'addresses and reduces visual bias' is not supported by Eq. (2).
  3. [Section III-A and III-C] The evaluation is circular in an important sense. The same rule-based pipeline generates the captions used for training, for retrieval ground truth, and for the VBM diagnostic. A low VBM may therefore simply indicate that the model aligns with the authors' own tag distribution (lane, motion, object class) rather than with human-meaningful temporal semantics. The paper should anchor the evaluation with independent human annotations or with a model trained on independently labeled captions.
  4. [Section III-A] The 3D detector and tracker used to generate all tracks are never named, and no accuracy or failure-mode analysis is given. Since tracking errors propagate directly into the training labels and the evaluation ground truth, every reported number depends on this unspecified component. Please specify the detector/tracker, report its performance (or use dataset ground-truth annotations where available), and assess how caption quality and VBM change with tracking noise.
  5. [Section IV-A, Table IV, Eqs. (3)-(8)] The pipeline depends on many hand-set thresholds (T_x, T_y, T_s, T_m, T_h, T_v, T_a, T_w, the lane yaw thresholds, and the VBM exclusion window), and no sensitivity analysis is provided. Because these thresholds fully determine the caption distribution, the paper should demonstrate that the main conclusions are robust to reasonable variations of these parameters.
minor comments (6)
  1. [Section III-B] The word 'extrtacted' should be 'extracted'.
  2. [Table II] The header repeats 'Differs in motion tag' for both columns; the second column should read 'Differs in lane tag'.
  3. [Figure 5, Eq. (2)] The temporal exclusion window used in the VBM computation is not specified, and the figure shows no error bars or significance information.
  4. [Figure 8 caption] The caption refers to 'Figure 1a and Figure 1b'; these should be Figure 8a and Figure 8b.
  5. [Section IV-B] The claim that evaluation uses the 'entire Waymo and NuScenes datasets' is ambiguous: it should state whether captions are generated by the same detector/tracker pipeline or taken from dataset ground truth, and how the candidate caption pool is constructed.
  6. [Table V] The candidate pool size for the baseline best-match selection is not given; report it for each dataset so the reader can gauge the difficulty of the retrieval task.

Circularity Check

0 steps flagged · score 0.0 of 10

No substantial circularity: the central claims are empirical, benchmarked against external models and cross-dataset zero-shot transfer; VBM validity is a measurement concern, not a definitional reduction.

full rationale

The paper's derivation chain is not circular. Captions are generated from LiDAR tracks by a rule-based system (Section III-A), and SwinBERT is trained on front-camera frames to predict those captions (Section III-B). The evaluation is anchored to external, unmodified baselines (InternVideo, ViCLIP, CLIP, VGG19) and to zero-shot transfer on Waymo and NuScenes with different camera configurations, so the headline captioning and retrieval results are externally falsifiable rather than forced by construction. The Visual Bias Measure (Eq. 2) is a newly introduced operational metric, defined as the relative BLEU4 drop when temporally nearby videos are excluded from retrieval; interpreting lower VBM as reduced visual bias is an assumption about the metric's validity, not a derivation in which the predicted quantity equals an input by construction. The same template captions serve as training labels and as retrieval ground truth, but the model must generalize to unseen datasets and clips, and the baselines were not trained on those captions, so the evaluation does not reduce to reproducing the training distribution. Concerns about VBM not being validated against human judgments or downstream tasks, and about the unnamed LiDAR detector/tracker, are correctness and robustness risks, not circularity. There are no load-bearing self-citations or imported uniqueness theorems.

Assumptions & free parameters 7 free parameters · 5 assumptions · 1 invented entities

The central claim depends on a dozen hand-set thresholds that determine every caption, an unnamed tracker whose errors become labels, and a new unvalidated metric. None of these are derived, calibrated, or swept, which is the main reason the evidence is weaker than the claims.

free parameters (7)
  • Neighborhood thresholds Tx, Ty = Tx: 30 m (pedestrian), 40 m (car/truck/bike); Ty: 15 m (pedestrian), 20 m (car/truck/bike)
    Table IV. Define which objects receive captions; hand-set, no sensitivity analysis.
  • Stationary threshold Ts = 0.1 (pedestrian/bike), 0.15 (car/truck)
    Table IV, Eq. (5). Decides which objects are treated as moving and therefore captioned.
  • Motion tag threshold Tm = 0.01 (pedestrian), 0.1 (car/truck/bike)
    Table IV. Binarizes relative distance change into approach/away/constant.
  • Lane lateral threshold Th = 2.5
    Table IV, Eq. (7). Sets lane-tag boundaries in the host frame.
  • Host velocity/acceleration/yaw-rate thresholds Tv, Ta, Tw = not reported numerically
    Eqs. (3)-(4). Shape all host captions and clip segmentation; values are deferred but never given in the text or appendix.
  • Baseline lane yaw thresholds phi_ol, phi_ou, phi_ll, phi_lu = not reported numerically
    Eq. (6). Assign oncoming/lateral/ongoing baseline lane tags; values are not specified.
  • VBM temporal exclusion window = not specified
    Section III-C says videos within a "predefined time window" are excluded; the window size is not given, and VBM numbers depend on it.
assumptions (5)
  • domain assumption Lane position is determinable from yaw angle and lateral position thresholds, Eqs. (6)-(7).
    The entire lane-tag system rests on hand-set yaw and lateral thresholds with no validation against maps or human lane labels.
  • domain assumption Neighbor motion is adequately described by the sign and magnitude of the lateral relative velocity, Eq. (8).
    Approach/away/constant/stationary are the only motion distinctions; subtle dynamics such as yielding, speed, or turning rate are discarded.
  • ad hoc to paper BLEU4 drop under temporal exclusion (VBM) quantifies visual bias and temporal understanding, Eq. (2).
    VBM is introduced in this paper and never validated against human judgment or an external benchmark.
  • domain assumption The LiDAR 3D detector and tracker produce accurate tracks.
    Section III-A invokes an unnamed "SOTA" pipeline; track errors propagate into captions that are both training labels and evaluation ground truth.
  • domain assumption Single-sentence captions from a 63-sentence pool suffice to learn temporal understanding.
    Section IV-A restricts training to length-1 captions, while generalization claims target two-sentence captions.
invented entities (1)
  • Visual Bias Measure (VBM)
    purpose: Quantifies how much retrieval quality drops when temporally nearby videos are excluded; lower values are claimed to indicate reduced visual bias and stronger temporal understanding.
    Defined in Eq. (2) and used as the primary evidence for the paper's headline claim. No external validation is provided, and the paper is the sole source of evidence for what VBM measures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temporal Object Captioning for Street Scene Videos from LiDAR Tracks." pith.science (2026). https://pith.science/paper/6EPXRTO3

@misc{pith2026250516594,
  author       = {Pith},
  title        = {Pith review of: Temporal Object Captioning for Street Scene Videos from LiDAR Tracks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6EPXRTO3}},
  note         = {Machine review of arXiv:2505.16594}
}
read the original abstract

Video captioning models have seen notable advancements in recent years, especially with regard to their ability to capture temporal information. While many research efforts have focused on architectural advancements, such as temporal attention mechanisms, there remains a notable gap in understanding how models capture and utilize temporal semantics for effective temporal feature extraction, especially in the context of Advanced Driver Assistance Systems. We propose an automated LiDAR-based captioning procedure that focuses on the temporal dynamics of traffic participants. Our approach uses a rule-based system to extract essential details such as lane position and relative motion from object tracks, followed by a template-based caption generation. Our findings show that training SwinBERT, a video captioning model, using only front camera images and supervised with our template-based captions, specifically designed to encapsulate fine-grained temporal behavior, leads to improved temporal understanding consistently across three datasets. In conclusion, our results clearly demonstrate that integrating LiDAR-based caption supervision significantly enhances temporal understanding, effectively addressing and reducing the inherent visual/static biases prevalent in current state-of-the-art model architectures.

Figures

Figures reproduced from arXiv: 2505.16594 by the authors.

Figure 1
Figure 1. LiDAR-based caption generation and video captioning model training. This figure shows the overall workflow of our approach. Object tracks extracted from raw LiDAR data using a SOTA 3D detector and tracker are used to identify event boundaries and generate object-level captions. The object caption and associated front camera RGB images (using event boundaries) are used to train a video captioning model (multi-modal m… view at source ↗
Figure 2
Figure 2. LiDAR-based caption generation and video captioning model training. This figure illustrates our overall caption generation procedure and the training process for the video captioning model. Using the host vehicle’s sensor data (speed and yaw rate) and object tracks from LiDAR data, we generate the object, lane and motion tags. These tags are used in the corresponding placeholders of the template sentence to construc… view at source ↗
Figure 4
Figure 4. Thresholds for lane tag. This figure illustrates the threshold values used in our neighbor captioning process. On the left, we depict the yaw angle thresholds applied to bounding boxes for lane tagging. On the right, we show the thresholds used for determining lane tags based on bounding box center positions. TABLE IV THRESHOLD VALUES FOR NEIGHBOR CAPTION CONSTRUCTION. THIS TABLE LISTS THE THRESHOLD VALUES USED TO A… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: shows the reduction in VBM achieved using our captions. Despite InternVideo and VideoCLIP being designed to enhance temporal understanding (e.g., through the Video￾MAE training strategy), our approach consistently exhibits the lowest VBM across all datasets. On our pro…
Figure 6
Figure 6. Figure 6: Two-sentence captions retrieval. The violin plots show the distri￾bution of BLEU4 scores between the ground truth captions of query and retrieved videos. A wider spread towards higher BLEU4 scores indicates better retrieval quality. The thick black line shows the mean …
Figure 7
Figure 7. Figure 7: UMAP representation of model’s embeddings. This figure illustrates the semantic structure of video embeddings produced by our model (left plot) and ViCLIP (right plot). Each point represents a video clip embedding, color￾coded by its associated lane tag. denser distrib…
Figure 8
Figure 8. Figure 8: Video-to-video retrieval. This figure shows an example of retrieving the top-three nearest neighbors for a given query video. The first column shows the query frame, and the second, third, and fourth columns present the first, second, and third nearest neighbor frames,…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 17 canonical work pages

  1. [1]

    Mammut: A simple architecture for joint learning for multimodal tasks,

    W. Kuo, A. Piergiovanni, D. Kim, X. Luo, B. Caine, W. Li, A. Ogale, L. Zhou, A. Dai, Z. Chen, C. Cui, and A. Angelova, “Mammut: A simple architecture for joint learning for multimodal tasks,” 2023. [Online]. Available: https://arxiv.org/abs/2303.16839

    arXiv 2023

  2. [2]

    Vlab: Enhancing video language pre- training by feature adapting and blending,

    X. He, S. Chen, F. Ma, Z. Huang, X. Jin, Z. Liu, D. Fu, Y . Yang, J. Liu, and J. Feng, “Vlab: Enhancing video language pre- training by feature adapting and blending,” 2023. [Online]. Available: https://arxiv.org/abs/2305.13167

    arXiv 2023

  3. [3]

    Valor: Vision-audio-language omni-perception pretraining model and dataset,

    S. Chen, X. He, L. Guo, X. Zhu, W. Wang, J. Tang, and J. Liu, “Valor: Vision-audio-language omni-perception pretraining model and dataset,”

    work page

  4. [4]

    Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics

    J. Wang, J. Jiao, L. Bao, S. He, Y . Liu, and W. Liu, “Self- supervised spatio-temporal representation learning for videos by predicting motion and appearance statistics,” 2019. [Online]. Available: https://arxiv.org/abs/1904.03597

    work page Pith review arXiv 2019

  5. [5]

    Unsupervised Pre-Training of Image Features on Non-Curated Data

    M. Caron, P. Bojanowski, J. Mairal, and A. Joulin, “Unsupervised pre-training of image features on non-curated data,” 2019. [Online]. Available: https://arxiv.org/abs/1905.01278

    work page Pith review arXiv 2019

  6. [6]

    Melanoma thick- ness prediction based on convolutional neural network with vgg-19 model transfer learning,

    J. Jaworek-Korjakowska, P. Kleczek, and M. Gorgon, “Melanoma thick- ness prediction based on convolutional neural network with vgg-19 model transfer learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 0–0

    work page 2019

  7. [7]

    Pre-training on grayscale imagenet improves medical image classification,

    Y . Xie and D. Richmond, “Pre-training on grayscale imagenet improves medical image classification,” inProceedings of the European confer- ence on computer vision (ECCV) workshops, 2018, pp. 0–0

    work page 2018

  8. [8]

    Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning

    T. Jin, S. Huang, Y . Li, and Z. Zhang, “Low-rank hoca: Efficient high-order cross-modal attention for video captioning,”arXiv preprint arXiv:1911.00212, 2019

    work page Pith review arXiv 1911

Show all 43 references
  1. [9]

    Sensor-augmented egocentric- video captioning with dynamic modal attention,

    K. Nakamura, H. Ohashi, and M. Okada, “Sensor-augmented egocentric- video captioning with dynamic modal attention,” inProceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 4220– 4229

    2021

  2. [10]

    Boosting video captioning with dynamic loss network,

    N. Ullah and P. P. Mohanta, “Boosting video captioning with dynamic loss network,”arXiv preprint arXiv:2107.11707, 2021

    2021 arXiv

  3. [11]

    Panoptic segmentation,

    A. Kirillov, K. He, R. Girshick, C. Rother, and P. Doll ´ar, “Panoptic segmentation,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 9404–9413

    2019

  4. [12]

    An empirical study of context in object detection,

    S. K. Divvala, D. Hoiem, J. H. Hays, A. A. Efros, and M. Hebert, “An empirical study of context in object detection,” in2009 IEEE Conference on computer vision and Pattern Recognition. IEEE, 2009, pp. 1271– 1278

    2009

  5. [13]

    End-to-end object detection with transformers,

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision. Springer, 2020, pp. 213– 229

    2020

  6. [14]

    Internvideo: General video foundation models via gen- erative and discriminative learning,

    Y . Wang, K. Li, Y . Li, Y . He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y . Liu, Z. Wanget al., “Internvideo: General video foundation models via gen- erative and discriminative learning,”arXiv preprint arXiv:2212.03191, 2022

    2022 arXiv

  7. [15]

    Internvideo2: Scaling video foundation models for multimodal video understanding,

    Y . Wang, K. Li, X. Li, J. Yu, Y . He, G. Chen, B. Pei, R. Zheng, J. Xu, Z. Wanget al., “Internvideo2: Scaling video foundation models for multimodal video understanding,”arXiv preprint arXiv:2403.15377, 2024

    2024 arXiv

  8. [16]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms,

    Z. Cheng, S. Leng, H. Zhang, Y . Xin, X. Li, G. Chen, Y . Zhu, W. Zhang, Z. Luo, D. Zhaoet al., “Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms,”arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  9. [17]

    Swinbert: End-to-end transformers with sparse attention for video captioning,

    K. Lin, L. Li, C.-C. Lin, F. Ahmed, Z. Gan, Z. Liu, Y . Lu, and L. Wang, “Swinbert: End-to-end transformers with sparse attention for video captioning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 949–17 958

    2022

  10. [18]

    Lingoqa: Visual question answering for autonomous driving

    A.-M. Marcu, L. Chen, J. H ¨unermann, A. Karnsund, B. Hanotte, P. Chidananda, S. Nair, V . Badrinarayanan, A. Kendall, J. Shottonet al., “Lingoqa: Visual question answering for autonomous driving.”

  11. [19]

    Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario,

    T. Qian, J. Chen, L. Zhuo, Y . Jiao, and Y .-G. Jiang, “Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 5, 2024, pp. 4542–4550

    2024

  12. [20]

    Nuscenes-mqa: Integrated evaluation of captions and qa for autonomous driving datasets using markup annotations,

    Y . Inoue, Y . Yada, K. Tanahashi, and Y . Yamaguchi, “Nuscenes-mqa: Integrated evaluation of captions and qa for autonomous driving datasets using markup annotations,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 930–938

    2024

  13. [21]

    Language prompt for autonomous driving,

    D. Wu, W. Han, T. Wang, Y . Liu, X. Zhang, and J. Shen, “Language prompt for autonomous driving,”arXiv preprint arXiv:2309.04379, 2023

    2023 arXiv

  14. [22]

    Covla: Comprehensive vision-language-action dataset for autonomous driving,

    H. Arai, K. Miwa, K. Sasaki, Y . Yamaguchi, K. Watanabe, S. Aoki, and I. Yamamoto, “Covla: Comprehensive vision-language-action dataset for autonomous driving,”arXiv preprint arXiv:2408.10845, 2024

    2024

  15. [23]

    nuscenes: A multimodal dataset for autonomous driving,

    H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11 621–11 631

    2020

  16. [24]

    Scalability in perception for autonomous driving: Waymo open dataset,

    P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V . Patnaik, P. Tsui, J. Guo, Y . Zhou, Y . Chai, B. Caineet al., “Scalability in perception for autonomous driving: Waymo open dataset,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020...

    2020

  17. [25]

    Kitti-360: A novel dataset and bench- marks for urban scene understanding in 2d and 3d,

    Y . Liao, J. Xie, and A. Geiger, “Kitti-360: A novel dataset and bench- marks for urban scene understanding in 2d and 3d,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 3, pp. 3292– 3310, 2022

    2022

  18. [26]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    D. Alexey, “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv: 2010.11929, 2020

    2010 arXiv

  19. [27]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

    2021

  20. [28]

    Training data-efficient image transformers & distillation through attention,

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J ´egou, “Training data-efficient image transformers & distillation through attention,” inInternational conference on machine learning. PMLR, 2021, pp. 10 347–10 357

    2021

  21. [29]

    Is space-time attention all you need for video understanding?

    G. Bertasius, H. Wang, and L. Torresani, “Is space-time attention all you need for video understanding?” inICML, vol. 2, no. 3, 2021, p. 4

    2021

  22. [30]

    Vivit: A video vision transformer,

    A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lu ˇci´c, and C. Schmid, “Vivit: A video vision transformer,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 6836–6846

    2021

  23. [31]

    Tuber: Tubelet transformer for video action detection,

    J. Zhao, Y . Zhang, X. Li, H. Chen, B. Shuai, M. Xu, C. Liu, K. Kundu, Y . Xiong, D. Modoloet al., “Tuber: Tubelet transformer for video action detection,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 13 598–13 607

    2022

  24. [32]

    End-to-end video instance segmentation with transformers,

    Y . Wang, Z. Xu, X. Wang, C. Shen, B. Cheng, H. Shen, and H. Xia, “End-to-end video instance segmentation with transformers,” inPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 8741–8750

    2021

  25. [33]

    Video-llama: An instruction-tuned audio-visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio-visual language model for video understanding,”arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  26. [34]

    Video-llava: Learning united visual representation by alignment before projection,

    B. Lin, Y . Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan, “Video-llava: Learning united visual representation by alignment before projection,” arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  27. [35]

    Adapt: Action-aware driving caption transformer,

    B. Jin, X. Liu, Y . Zheng, P. Li, H. Zhao, T. Zhang, Y . Zheng, G. Zhou, and J. Liu, “Adapt: Action-aware driving caption transformer,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 7554–7561

    2023

  28. [36]

    Textual explanations for self-driving vehicles,

    J. Kim, A. Rohrbach, T. Darrell, J. Canny, and Z. Akata, “Textual explanations for self-driving vehicles,” inProceedings of the European conference on computer vision (ECCV), 2018, pp. 563–578

    2018

  29. [37]

    Internvid: A large-scale video-text dataset for multi- modal understanding and generation,

    Y . Wang, Y . He, Y . Li, K. Li, J. Yu, X. Ma, X. Li, G. Chen, X. Chen, Y . Wanget al., “Internvid: A large-scale video-text dataset for multi- modal understanding and generation,”arXiv preprint arXiv:2307.06942, 2023

    2023 arXiv

  30. [38]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  31. [39]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  32. [40]

    Can masking back- ground and object reduce static bias for zero-shot action recognition?

    T. Fukuzawa, K. Hara, H. Kataoka, and T. Tamaki, “Can masking back- ground and object reduce static bias for zero-shot action recognition?” inInternational Conference on Multimedia Modeling. Springer, 2025, pp. 366–379

    2025

  33. [41]

    Another efficient algorithm for convex hulls in two dimensions,

    A. M. Andrew, “Another efficient algorithm for convex hulls in two dimensions,”Information Processing Letters, vol. 9, no. 5, pp. 216–219, 1979

    1979

  34. [42]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inProceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318. VI. SUPPLEMENTARYMATERIAL In the next section, we det...

    2002

  35. [2023]

    Available: https://arxiv.org/abs/2304.08345

    [Online]. Available: https://arxiv.org/abs/2304.08345

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.