Pith. sign in

REVIEW 3 major objections 6 minor 51 references

Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adding human-readable textual reasons to a vision pipeline improves pedestrian intent prediction, reporting 87.6% accuracy on PIE++ and 95.4% on JAAD.

desk verdict Useful new reason-annotated dataset, but the central claim that reasons improve intent is untested because the reason modality never enters the model as input. read the letter →

arxiv 2411.13302 v1 pith:UJHXGHOD submitted 2024-11-20 cs.CV

Vaishnavi Khindkar , Vineeth Balasubramanian , Chetan Arora , Anbumani Subramanian , C.V. Jawahar This is my paper · ORCID
classification cs.CV
keywords pedestrianintentpredictioncross-modalrepresentationlearningtextualreasonsPIE++datasetmulti-taskautonomousdrivingexplainability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that knowing why a pedestrian might cross helps predict whether they will cross. The authors build PIE++, a reason-enriched version of the PIE dataset with multi-label textual explanations for each pedestrian's intent, and propose MINDREAD, a multi-task model that fuses video features with learned embeddings of these reasons. Their experiments report that MINDREAD improves intent accuracy by 5.6% and F1 by 7% over the previous state of the art on PIE++, reaching 87.6% accuracy, and improves accuracy by 4.4% on JAAD. If the claim holds, both predictive accuracy and human interpretability of autonomous-driving systems improve, because the model can state a reason alongside its crossing decision.

What carries the argument

The load-bearing component is the Correlated Semantic Explanation Affinity (CSEA) module, which treats the textual reasons as nodes in a directed graph whose edges are co-occurrence probabilities, then applies a graph convolutional network to produce correlated semantic embeddings. A transformer-based feature encoder (TFE) extracts spatial and temporal visual features from local context, global context, and bounding boxes, and an attention mechanism fuses the two representations into a shared cross-modal vector. This fused vector feeds two prediction heads, one for intent and one for multi-label reasons. The paper's hypothesis is that reason co-occurrence structure, captured by the graph, is what lets the language modality contribute beyond simple label information.

What would settle it

A concrete test is to collect reason annotations without revealing the crossing label to annotators and then retrain MINDREAD on that version. If intent accuracy no longer improves over the vision-only baseline (or the gain falls to the level of a model trained with a random auxiliary head), the paper's central claim would be refuted; the same test could be simulated by scrambling the reason-to-intent association in PIE++ and checking whether the 4.2% cross-modality gain persists.

Watch

Extended reading notes

Core claim

The central claim is that the reason behind a pedestrian's intent is not just an explainability add-on but a useful learning signal: a cross-modal representation that joins spatiotemporal visual features with textual reason embeddings predicts the binary cross/no-cross intent better than vision-only baselines. On the PIE++ dataset the proposed MINDREAD model attains 87.6% accuracy and 95.0% F1, compared with 82.0% and 88.0% for the prior CIA method, and the reason-prediction head itself reaches 72.4% accuracy. Ablating the cross-modal fusion drops intent accuracy by 4.2%, which the authors take as evidence that the textual reasons carry information not present in the visual stream. On JAAD, enriched with the same reason categories, MINDREAD reaches 95.4% intent accuracy versus 91.0% for PV-LSTM.

Load-bearing premise

The weakest premise is that the PIE++ reason annotations are independent signals for intent, even though annotators were shown the ground-truth crossing label before choosing reasons; if the reasons merely restate the intent label, the reported gains could come from the multi-task label embedding rather than from human-understandable reasoning.

Editorial extensions

If this is right

  • Systems that predict pedestrian intent can output a human-readable reason alongside the binary decision, without sacrificing accuracy.
  • On the benchmarks used, the gains are consistent across both PIE++ and JAAD, suggesting the approach transfers across datasets.
  • The PIE++ annotations provide a resource for training and evaluating reason-based intent models.
  • Training a model to predict reasons jointly with intent yields better intent prediction than training intent alone, per Table VIII.
  • The method's runtime is comparable to or faster than a prior baseline (4.3 ms vs 4.7 ms on JAAD), so the cross-modal fusion does not obviously cost latency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A decisive test of the causal claim would re-collect reasons without showing annotators the ground-truth crossing label; if the accuracy gain shrinks, the current improvements may largely reflect an indirect encoding of the intent label through the multi-task head.
  • If reasons are truly informative, the same cross-modal recipe could be extended to other vulnerable road users (cyclists, motorcyclists) and to finer-grained intent distinctions such as crossing start time, which the paper does not test.
  • The reason taxonomy itself could be made richer (gaze direction, phone use, vehicle approach speed) and the affinity graph would then need to be re-learned; the paper's current graph is tied to its 17 reason categories.
  • Because the model outputs reasons, one could audit failures by checking whether the predicted reason is consistent with the predicted intent; such consistency checks are not reported in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces PIE++, an extension of the PIE dataset with multi-label textual reasons for pedestrian crossing/no-crossing intent, and proposes MINDREAD, a multi-task framework that fuses visual spatiotemporal features with learned reason embeddings via an attention mechanism. The authors report significant improvements over prior methods on PIE++ and JAAD, and provide ablations, user studies, and runtime analysis. The paper's central hypothesis is that predicting the "WHY" can improve prediction of the "WHAT," i.e., that human-understandable reasons improve pedestrian intent estimation.

Significance. If the central claim is sustained, this is a novel and practically relevant contribution: it would be the first to use textual reasons to improve pedestrian intent prediction while also providing explainable outputs. The dataset (PIE++) is a potentially useful resource, and the user study adds evidence for the utility of reasons. However, the current experimental design does not actually test the claim because the reason modality is never used as input to the model; the reported gains are consistent with a multi-task label-embedding effect rather than with the semantic content of reasons. The annotation protocol also introduces a circularity risk. These issues must be addressed before the paper can be accepted.

major comments (3)
  1. [Sec. IV-C, Eq. (C = F · X^T)] The cross-modal representation is defined as C = F · X^T, where F is the visual attention output and X is the learned correlated semantic explanation embedding matrix from CSEA. X is a function only of the fixed set of reason categories and their co-occurrence statistics; it is not conditioned on the input video or on any per-pedestrian reason observation. Therefore, at both training and inference, the model receives no textual reason as input. The improvement in Table VIII (83.4% vs 87.6% intent accuracy) demonstrates that adding a parameterized reason-label embedding pathway helps, but it does not test whether human-understandable reasons improve intent prediction. To support the central claim, the model must ingest reasons as information (e.g., by conditioning on ground-truth or predicted per-instance reason labels), or the claims must be reframed to a multi-task label-embedding study.
  2. [Sec. III, 'Sourcing Reason Annotations in PIE++'] Annotators were provided with the ground-truth pedestrian intent annotations before selecting reasons. This makes the reason labels partly derived from the intent labels, so the reasons are not an independent signal for intent. The paper's hypothesis is that reasons help predict intent; if reasons are produced by annotators who already know the intent, the reported gains may be an artifact of the reasons re-encoding the intent labels. Please either collect reasons without revealing intent (or with a separate set of annotators), or provide an analysis showing that reasons carry information beyond the intent label (e.g., measure the mutual information between reasons and intent, or train the reason head on a subset without intent supervision).
  3. [Table V, SOTA comparison] MINDREAD uses a Swin-V2-L + Transformer backbone, while the baseline methods (PIEint, STIP, CIA) use VGG + GRU. Table XI shows that replacing VGG + GRU with Swin-V2 + Transformer in MINDREAD improves intent accuracy by 0.6% (87.0% vs 87.6%). Since the reported improvement over CIA is 5.6%, the backbone difference is a confound in the SOTA comparison. Please provide results where all methods use the same backbone, or at least quantify the contribution of the backbone to the reported gains.
minor comments (6)
  1. [Table IV] The user study is based on only 5 subjects; please report the raw counts and discuss the limitations of this small sample size.
  2. [Sec. I] The claim that this is the "first pedestrian reason+intent prediction dataset" is not fully substantiated; the PSI dataset [28] is cited but not compared as a reason-bearing dataset. Please clarify the distinction or discuss PSI explicitly.
  3. [Table VI] The reason prediction accuracy metric is not defined; please specify whether it is exact-match multi-label accuracy, Hamming accuracy, or another metric, and state how it is computed.
  4. [Sec. VI, 'MINDREAD without Cross-modality'] The name of this ablation suggests that the variant uses no text modality, but it still trains a reason classification head; please clarify that this ablation removes the CSEA module and the cross-modal fusion, rather than removing all text-related processing.
  5. [Sec. IV-A] The adjacency matrix A is computed from co-occurrence of reasons "within the dataset"; please clarify whether these statistics are computed on the training split only to avoid information leakage through the validation/test splits.
  6. [Sec. IV-C] Please provide the dimensions of the tensors in the cross-modal fusion (F, X, and C) to make the operation concrete and reproducible.

Circularity Check

2 steps flagged · score 6.0 of 10

The central claim that reasons improve intent is circular: PIE++ reason labels are annotated using ground-truth intent, and the cross-modal pathway uses a fixed label-embedding matrix, so the reported gains reduce to multi-task label re-encoding.

  1. self definitional [Section III, 'Sourcing Reason Annotations in PIE++' and Tables II-III]
    "Before obtaining the multi-label reason annotations in PIE++, we provided videos for all the 1842 pedestrians in PIE to 5 annotation professionals with experience in data annotation in the mobility industry. They were provided with all relevant information, including pedestrian intent annotations and pedestrian attributes (e.g. looking, walking, standing) to help decide on the plausible reasons leading to the pedestrian's intent in a given scene."

    Annotators were given the ground-truth intent before choosing reasons, and Tables II/III assign every reason to a single intent class ('Cross' or 'No-Cross'). Hence the reason labels are a function of the intent label by construction. Claiming that 'predicting the WHY helps understand the WHAT' is then circular: the auxiliary reason head re-encodes the intent label, so the gains in Tables V-VIII reflect multi-task label re-encoding rather than independent semantic content of reasons.

  2. fitted input called prediction [Section IV-C, cross-modal representation C = F · X^T; Section IV-A (CSEA)]
    "Finally, our cross-modal representation is given by, C = F · X^T, where F represents final attention-based feature representations and X represents the learned correlated semantic explanation embeddings."

    X is the CSEA output: a GCN embedding of the fixed set of reason texts and their dataset co-occurrence statistics. It is a constant matrix across all test inputs; it is not conditioned on the current video or on any per-pedestrian reason observation. Therefore the 'language modality' never enters the model as text; C=F·X^T is just a learned bilinear projection of visual features through a label-embedding matrix. The Table VIII cross-modality gain therefore demonstrates that an extra parameterized label-embedding pathway helps, not that textual reasons improve intent prediction. Since the reason labels were themselves derived from intent (Step 1), this is a fitted-input artifact renamed as cross-modal reasoning.

full rationale

The paper is self-contained against external benchmarks and does not rely on load-bearing self-citations; its references are to independent prior work. The central difficulty is internal. The PIE++ reason annotations are sourced by showing annotators the ground-truth intent (Sec III), and the reason vocabulary is partitioned by intent class (Tables II and III). Consequently, the 'WHY' labels are definitionally tied to the 'WHAT' label, so the paper's main result—that adding reasons improves intent prediction—is not an independent test of the hypothesis. In addition, the cross-modal fusion C=F·X^T uses a fixed learned embedding matrix X (from CSEA) rather than any per-instance textual input; the model never receives a reason as input at inference. The reported improvements (5.6% accuracy and 7% F1 on PIE++; 4.4% on JAAD) are therefore consistent with a multi-task label-embedding effect rather than with human-understandable reasoning. This is a partial circularity: the central claim reduces, by dataset construction and by the fixed label-embedding equation, to re-encoding the intent label. I set the score to 6 rather than higher because the paper does contribute a new annotation resource and a strong visual backbone, and because no self-citation chain forces the result.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central results rest on a small number of hand-chosen and data-derived design elements: the fixed 17-class reason taxonomy, the co-occurrence matrix computed from the same annotations used for evaluation, and the loss weights. The method also assumes the binary intent definition from PIE and that pretrained text embeddings are semantically useful. No new physical entities are introduced.

free parameters (3)
  • Reason category set (17 classes) = 14 cross-related reasons and 3 no-cross reasons (Tables II and III)
    Chosen by hand after a user survey and inspection of PIE scenes; the label space shapes what the model can output and the co-occurrence graph learned from it.
  • Co-occurrence adjacency matrix A = Conditional probabilities P(E_j | E_i) estimated from PIE++ annotations
    Computed from the same dataset used for evaluation; the GCN reason embeddings and the cross-modal fusion depend on these counts.
  • Loss weights gamma_R, gamma_I = 1.0 and 1.0
    Selected from a small grid (0.5, 1.0); the paper reports that weighting one loss more than the other did not improve performance (Table XIII).
assumptions (4)
  • domain assumption Pedestrian intent is a binary cross/no-cross label and can be explained by a small fixed set of 17 multi-label reasons.
    Adopted from the PIE benchmark and the authors' three-stage category design; invoked throughout Sec III and IV.
  • domain assumption Sentence-BERT embeddings of reason phrases carry semantic content useful for intent estimation.
    The CSEA module uses pretrained Sentence-BERT embeddings as node features (Sec IV-A); no fine-tuning on domain text is described.
  • ad hoc to paper Annotators given the ground-truth crossing intent can still produce reasons that are valid and unbiased.
    Annotation protocol in Sec III reveals the intent label was provided to annotators before choosing reasons; the paper does not analyze how this affects independence.
  • standard math The cross-modal fusion via attention and GCN provides a correct mechanism for combining visual and textual signals.
    GCN, transformer, and attention operations are standard, but their adequacy here is an empirical assertion (Sec IV).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach." pith.science (2026). https://pith.science/paper/UJHXGHOD

@misc{pith2026241113302,
  author       = {Pith},
  title        = {Pith review of: Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJHXGHOD}},
  note         = {Machine review of arXiv:2411.13302}
}
read the original abstract

With the increased importance of autonomous navigation systems has come an increasing need to protect the safety of Vulnerable Road Users (VRUs) such as pedestrians. Predicting pedestrian intent is one such challenging task, where prior work predicts the binary cross/no-cross intention with a fusion of visual and motion features. However, there has been no effort so far to hedge such predictions with human-understandable reasons. We address this issue by introducing a novel problem setting of exploring the intuitive reasoning behind a pedestrian's intent. In particular, we show that predicting the 'WHY' can be very useful in understanding the 'WHAT'. To this end, we propose a novel, reason-enriched PIE++ dataset consisting of multi-label textual explanations/reasons for pedestrian intent. We also introduce a novel multi-task learning framework called MINDREAD, which leverages a cross-modal representation learning framework for predicting pedestrian intent as well as the reason behind the intent. Our comprehensive experiments show significant improvement of 5.6% and 7% in accuracy and F1-score for the task of intent prediction on the PIE++ dataset using MINDREAD. We also achieved a 4.4% improvement in accuracy on a commonly used JAAD dataset. Extensive evaluation using quantitative/qualitative metrics and user studies shows the effectiveness of our approach.

Figures

Figures reproduced from arXiv: 2411.13302 by the authors.

Figure 1
Figure 1. Illustration of our overall objectives. Given a sample scene from the PIE dataset [1], we study the usefulness of “WHY” (reason) for [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall architecture of MINDREAD (cross-Modal representatIon learNing moDel for REAsoning peDestrian-intent) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 41 canonical work pages

  1. [1]

    Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,

    A. Rasouli, I. Kotseruba, T. Kunic, and J. K. Tsotsos, “Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019, pp. 6262–6271

    work page 2019

  2. [2]

    Cou- pling intent and action for pedestrian crossing behavior prediction,

    Y . Yao, E. Atkins, M. J. Roberson, R. Vasudevan, and X. Du, “Cou- pling intent and action for pedestrian crossing behavior prediction,” International Joint Conferences on Artificial Intelligence (IJCAI) , 2021

    work page 2021

  3. [3]

    The Cityscapes Dataset for Semantic Urban Scene Understanding,

    M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Be- nenson, U. Franke, S. Roth, and B. Schiele, “The Cityscapes Dataset for Semantic Urban Scene Understanding,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    work page 2016

  4. [4]

    BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning,

    F. Yu, H. Chen, X. Wang, W. Xian, Y . Chen, F. Liu, V . Madhavan, and T. Darrell, “BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020

    work page 2020

  5. [5]

    Are we ready for autonomous driving? the kitti vision benchmark suite,

    A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 3354– 3361

    work page 2012

  6. [6]

    Vulnerable road users and connected autonomous vehicles interaction: A survey,

    A. Reyes-Mu ˜noz and J. Guerrero-Ib ´a˜nez, “Vulnerable road users and connected autonomous vehicles interaction: A survey,” Sensors, vol. 22, no. 12, p. 4614, 2022

    work page 2022

  7. [7]

    Vulnerable Road Users, Position/Policy Statement, NATIONAL SAFETY COUNCIL,

    “Vulnerable Road Users, Position/Policy Statement, NATIONAL SAFETY COUNCIL,” https://www.nsc.org/ getattachment/d5babee6-582d-4e66-804f-8d06f9b021a4/ t-vulnerable-road-users-147, 2022, [Online; accessed 10-November- 2022]

    work page 2022

  8. [8]

    VULNERABLE ROAD USER (VRU) PROTECTION,

    “VULNERABLE ROAD USER (VRU) PROTECTION,” https://www.euroncap.com/en/vehicle-safety/the-ratings-explained/ vulnerable-road-user-vru-protection/, 2022, [Online; accessed 10-November-2022]

    work page 2022

Show all 51 references
  1. [9]

    Spatiotemporal relationship reasoning for pedestrian intent prediction,

    B. Liu, E. Adeli, Z. Cao, K.-H. Lee, A. Shenoi, A. Gaidon, and J. C. Niebles, “Spatiotemporal relationship reasoning for pedestrian intent prediction,” IEEE Robotics and Automation Letters (R-AL) , vol. 5, no. 2, pp. 3485–3492, 2020

    2020

  2. [10]

    Intformer: Predicting pedestrian intention with the aid of the transformer architecture,

    J. Lorenzo, I. Parra, and M. Sotelo, “Intformer: Predicting pedestrian intention with the aid of the transformer architecture,” arXiv preprint arXiv:2105.08647, 2021

    2021 arXiv

  3. [11]

    Fussi-net: Fusion of spatio-temporal skeletons for intention prediction network,

    F. Piccoli, R. Balakrishnan, M. J. Perez, M. Sachdeo, C. Nunez, M. Tang, K. Andreasson, K. Bjurek, R. D. Raj, E. Davidsson, et al., “Fussi-net: Fusion of spatio-temporal skeletons for intention prediction network,” in 2020 54th Asilomar Conference on Signals, Systems, and Comp...

    2020

  4. [12]

    Context model for pedestrian intention prediction using factored latent-dynamic condi- tional random fields,

    S. Neogi, M. Hoy, K. Dang, H. Yu, and J. Dauwels, “Context model for pedestrian intention prediction using factored latent-dynamic condi- tional random fields,” IEEE Transactions on Intelligent Transportation Systems (T-ITS), 2020

    2020

  5. [13]

    Real-time intent prediction of pedestrians for autonomous ground vehicles via spatio-temporal densenet,

    K. Saleh, M. Hossny, and S. Nahavandi, “Real-time intent prediction of pedestrians for autonomous ground vehicles via spatio-temporal densenet,” in 2019 International Conference on Robotics and Automa- tion (ICRA). IEEE, 2019, pp. 9704–9710

    2019

  6. [14]

    Intent prediction of vulnerable road users from motion trajec- tories using stacked lstm network,

    ——, “Intent prediction of vulnerable road users from motion trajec- tories using stacked lstm network,” in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC) , 2017, pp. 327–332

    2017

  7. [15]

    Behavioral reasoning theory: Identifying new linkages underlying intentions and behavior,

    J. D. Westaby, “Behavioral reasoning theory: Identifying new linkages underlying intentions and behavior,” Organizational Behavior and Human Decision Processes , vol. 98, no. 2, pp. 97–120, 2005

    2005

  8. [16]

    Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior,

    A. Rasouli, I. Kotseruba, and J. K. Tsotsos, “Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior,” in Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV-W), 2017, pp. 206–213

    2017

  9. [17]

    Bifold and semantic reasoning for pedestrian behavior prediction,

    A. Rasouli, M. Rohani, and J. Luo, “Bifold and semantic reasoning for pedestrian behavior prediction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2021, pp. 15 600–15 610

    2021

  10. [18]

    Pedestrian action antici- pation using contextual feature fusion in stacked rnns,

    A. Rasouli, I. Kotseruba, and J. K. Tsotsos, “Pedestrian action antici- pation using contextual feature fusion in stacked rnns,” arXiv preprint arXiv:2005.06582, 2020

    2005 arXiv

  11. [19]

    Graph-sim: A graph-based spatiotemporal interaction mod- elling for pedestrian action prediction,

    T. Yau, S. Malekmohammadi, A. Rasouli, P. Lakner, M. Rohani, and J. Luo, “Graph-sim: A graph-based spatiotemporal interaction mod- elling for pedestrian action prediction,” in 2021 IEEE International Conference on Robotics and Automation (ICRA) , 2021, pp. 8580– 8586

    2021

  12. [20]

    Benchmark for evaluating pedestrian action prediction,

    I. Kotseruba, A. Rasouli, and J. K. Tsotsos, “Benchmark for evaluating pedestrian action prediction,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , January 2021, pp. 1258–1268

    2021

  13. [21]

    Social aware multi- modal pedestrian crossing behavior prediction,

    X. Zhai, Z. Hu, D. Yang, L. Zhou, and J. Liu, “Social aware multi- modal pedestrian crossing behavior prediction,” in Proceedings of the Asian Conference on Computer Vision , 2022, pp. 4428–4443

    2022

  14. [22]

    Multi-modal hybrid architecture for pedestrian action prediction,

    A. Rasouli, T. Yau, M. Rohani, and J. Luo, “Multi-modal hybrid architecture for pedestrian action prediction,” in 2022 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2022, pp. 91–97

    2022

  15. [23]

    Context-aware captions from context-agnostic supervision,

    R. Vedantam, S. Bengio, K. Murphy, D. Parikh, and G. Chechik, “Context-aware captions from context-agnostic supervision,” in Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 251–260

    2017

  16. [24]

    Grounding visual explanations,

    L. A. Hendricks, R. Hu, T. Darrell, and Z. Akata, “Grounding visual explanations,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 264–279

    2018

  17. [25]

    Show, attend and tell: Neural image caption generation with visual attention,

    K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y . Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in International Conference on Machine Learning (ICML) . PMLR, 2015, pp. 2048–2057

    2015

  18. [26]

    Deep learning,

    Y . LeCun, Y . Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015

    2015

  19. [27]

    Textual explanations for self-driving vehicles,

    J. Kim, A. Rohrbach, T. Darrell, J. Canny, and Z. Akata, “Textual explanations for self-driving vehicles,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 563–578

    2018

  20. [28]

    Psi: A pedestrian behavior dataset for socially intelligent autonomous car,

    T. Chen, R. Tian, Y . Chen, J. Domeyer, H. Toyoda, R. Sherony, T. Jing, and Z. Ding, “Psi: A pedestrian behavior dataset for socially intelligent autonomous car,” arXiv preprint arXiv:2112.02604 , 2021

    2021 arXiv

  21. [29]

    A peek into the reasoning of neural networks: Interpreting with structural visual concepts,

    Y . Ge, Y . Xiao, Z. Xu, M. Zheng, S. Karanam, T. Chen, L. Itti, and Z. Wu, “A peek into the reasoning of neural networks: Interpreting with structural visual concepts,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp....

    2021

  22. [30]

    Drive: Deep reinforced accident anticipation with visual explanation,

    W. Bao, Q. Yu, and Y . Kong, “Drive: Deep reinforced accident anticipation with visual explanation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2021, pp. 7619–7628

    2021

  23. [31]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al., “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning (ICML) . PMLR, 2021, pp. 8748– 8763

    2021

  24. [32]

    Multimodal contrastive training for visual representation learning,

    X. Yuan, Z. Lin, J. Kuen, J. Zhang, Y . Wang, M. Maire, A. Kale, and B. Faieta, “Multimodal contrastive training for visual representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2021, pp. 6995–7004

    2021

  25. [33]

    Shared cross-modal trajectory prediction for autonomous driving,

    C. Choi, J. H. Choi, J. Li, and S. Malla, “Shared cross-modal trajectory prediction for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 244–253

    2021

  26. [34]

    Pedestrian inten- tion prediction for autonomous driving using a multiple stakeholder perspective model,

    K. Kim, Y . K. Lee, H. Ahn, S. Hahn, and S. Oh, “Pedestrian inten- tion prediction for autonomous driving using a multiple stakeholder perspective model,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2020, pp. 7957–7962

    2020

  27. [35]

    Joint intention and trajectory prediction based on transformer,

    Z. Sui, Y . Zhou, X. Zhao, A. Chen, and Y . Ni, “Joint intention and trajectory prediction based on transformer,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2021, pp. 7082–7088

    2021

  28. [36]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” CoRR, vol. abs/1409.1556, 2015

    2015 arXiv

  29. [37]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation,

    K. Cho, B. van Merrienboer, C ¸ . G¨ulc ¸ehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” in Empirical Methods in Natural Language Processing (EMNLP) , 2014

    2014

  30. [38]

    Swin transformer v2: Scaling up capacity and resolution,

    Z. Liu, H. Hu, Y . Lin, Z. Yao, Z. Xie, Y . Wei, J. Ning, Y . Cao, Z. Zhang, L. Dong, F. Wei, and B. Guo, “Swin transformer v2: Scaling up capacity and resolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 12 0...

    2022

  31. [39]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 201...

    2019

  32. [40]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Ad- vances in Neural Information Processing Systems (NeurIPS) , vol. 30, 2017

    2017

  33. [41]

    Pedx: Bench- mark dataset for metric 3-d pose estimation of pedestrians in complex urban intersections,

    W. Kim, M. S. Ramanagopal, C. Barto, M.-Y . Yu, K. Rosaen, N. Goumas, R. Vasudevan, and M. Johnson-Roberson, “Pedx: Bench- mark dataset for metric 3-d pose estimation of pedestrians in complex urban intersections,” IEEE Robotics and Automation Letters (RA-L) , vol. 4, no. 2, p...

    1940

  34. [42]

    nuscenes: A multimodal dataset for autonomous driving,

    H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2020, pp. 1...

    2020

  35. [43]

    Intraclass correlations: uses in assessing rater reliability

    P. E. Shrout and J. L. Fleiss, “Intraclass correlations: uses in assessing rater reliability.” Psychological bulletin, vol. 86, no. 2, p. 420, 1979

    1979

  36. [44]

    Semi-supervised classification with graph convolutional networks,

    M. Welling and T. N. Kipf, “Semi-supervised classification with graph convolutional networks,” in J. International Conference on Learning Representations (ICLR), 2016

    2016

  37. [45]

    Imagenet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. , “Imagenet large scale visual recognition challenge,” International Journal of Computer Vision (IJCV) , vol. 115, no. 3, pp. 211–252, 2015

    2015

  38. [46]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly,et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations (ICLR), 2020

    2020

  39. [47]

    Learning deep transformer models for machine translation,

    Q. Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, and L. S. Chao, “Learning deep transformer models for machine translation,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) , 2019, pp. 1810–1822

    2019

  40. [48]

    Adaptive input representations for neural language modeling,

    A. Baevski and M. Auli, “Adaptive input representations for neural language modeling,” in International Conference on Learning Repre- sentations (ICLR), 2018, pp. 1810–1822

    2018

  41. [49]

    Effective approaches to attention-based neural machine translation,

    M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” inProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015, pp. 1412–1421

    2015

  42. [50]

    Pedestrian intention prediction: A multi-task perspective,

    S. A. Bouhsain, S. Saadatnejad, and A. Alahi, “Pedestrian intention prediction: A multi-task perspective,” arXiv preprint arXiv:2010.10270, 2020

    2010 arXiv

  43. [51]

    Glove: Global vectors for word representation,

    J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 2014, pp. 1532–1543

    2014

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.