Pith. sign in

REVIEW 4 major objections 5 minor 19 references

Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BF-PIP claims that zero-shot video prompts, combining raw temporal clips with speed and bounding-box cues, predict pedestrian crossing intention at 73% accuracy on JAAD-beh, beating a GPT-4V baseline by 18%.

desk verdict A post-hoc best-of-eight test-set selection and a mismatch in the reported improvement sink the 73% headline, but the video-input idea and the ablation study are worth a serious referee's time. read the letter →

arxiv 2507.21161 v1 pith:EACVMQQ5 submitted 2025-07-25 cs.CV cs.AIcs.LG

Pallavi Zambare , Venkata Nikhil Thanikella , Ying Liu This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords pedestrianintentionpredictionzero-shotlearningmultimodallargelanguagemodelsvideounderstandingautonomousdrivingGemini2.5ProJAADbenchmarkpromptengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BF-PIP asks whether a general multimodal model can infer pedestrian crossing intent from short video clips without any task-specific training. The paper claims it can: on the JAAD-beh benchmark, a prompt built around a 16-frame raw video clip plus ego-vehicle speed lets Gemini 2.5 Pro reach 73% prediction accuracy, 18% above a GPT-4V frame-based baseline and 6% above the prior multimodal baseline OmniPredict. The point of caring is that autonomy pipelines normally need supervised retraining to handle new scenes; if this holds, intention prediction becomes a prompt-engineering problem rather than a model-training problem. The authors also find that raw unannotated video alone reaches 65% accuracy, so temporal continuity itself carries much of the signal.

What carries the argument

The load-bearing object is the BF-PIP multimodal prompt pipeline. It takes a 16-frame video clip (about 0.5 seconds at 30 FPS), optional rendered bounding-box overlays from JAAD, ego-vehicle speed, and a two-stage instruction prompt that assigns the model an observer role and requests a single-word binary decision with chain-of-thought reasoning. The pipeline runs deterministically (temperature 0, fixed seed) and makes five predictions per instance. What carries the argument is the contrast between this temporally continuous input and the still-frame inputs of prior MLLM baselines: motion, hesitation, and gaze shifts are visible to the model rather than reconstructed from sparse images.

What would settle it

Run the same prompt pipeline on the same JAAD-beh test clips but replace the ground-truth rendered boxes with boxes from a real-time detector (e.g., YOLO) and measure accuracy; if the result falls toward the 0.65 unannotated-video level, the 0.73 annotated-mode result is not robust to detection error.

Watch

Extended reading notes

Core claim

The central claim is that continuous temporal video, rather than discrete frames, is what lets a zero-shot multimodal large language model anticipate pedestrian crossing behavior. Concretely, BF-PIP feeds Gemini 2.5 Pro a 16-frame clip sampled 30 frames before the crossing event, optionally renders the JAAD ground-truth bounding box onto each frame, and adds the ego-vehicle's speed class in a structured prompt. In the annotated-video-plus-speed configuration the model reaches 0.73 accuracy, 0.80 F1, and 0.96 precision on JAAD-beh, outperforming a GPT-4V frame baseline by 18% and the prior MLLM method OmniPredict by 6%, as well as several supervised sequence models. The ablation attributes the gain to combining visual annotation with speed: adding explicit bounding-box coordinate text to video actually hurt accuracy, while speed consistently helped.

Load-bearing premise

The headline 73% result is measured with JAAD's ground-truth bounding boxes rendered into the clips, so the method assumes a deployed system can supply accurate per-pedestrian boxes at prediction time; with imperfect detector boxes the reported accuracy would likely drop.

Editorial extensions

If this is right

  • If BF-PIP's 73% accuracy holds, pedestrian crossing prediction at a 1-second horizon can be done without training data or fine-tuning, using only a short video clip and speed metadata.
  • The reported advantage over the frame-based GPT-4V baseline assigns a concrete value to temporal continuity: replacing still frames with raw video changes decision quality on the same benchmark.
  • Ego-vehicle speed is a cheap, reliable signal; the ablation shows adding speed to annotated video raises accuracy from 0.64 to 0.73, so speed should be retained in future prompt designs.
  • Raw unannotated video alone reaches 0.65 accuracy, meaning even without bounding boxes the model captures a large share of the crossing signal; this is the deployment-relevant number if no detector is available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 0.73 result is an upper bound for real deployments that detect rather than annotate pedestrians: replacing ground-truth JAAD boxes with detector outputs would likely land between the 0.73 annotated-video-plus-speed and 0.65 unannotated numbers.
  • The ablation's counterintuitive result that raw coordinate text hurts performance suggests the model benefits from visually grounded boxes but cannot consume them as numbers; a natural extension is testing spatial-language descriptions ('the pedestrian is on the left sidewalk near the crosswalk') instead of $(x,y,w,h)$ tuples.
  • Because the prompt is dataset-agnostic, the same video-plus-speed recipe could be tested on newer pedestrian datasets (e.g., PIE) or on cyclists and scooters, which would tell whether the temporal-continuity advantage transfers beyond JAAD's distribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes BF-PIP, a zero-shot pedestrian crossing intention prediction method built on Gemini 2.5 Pro. The method takes short video clips from the JAAD-beh dataset, optionally enriched with rendered bounding boxes (annotated mode) or raw coordinate values, plus ego-vehicle speed, and uses a hand-crafted two-stage prompt to produce a binary cross/not-cross prediction. Evaluation on the JAAD-beh test set reports a best accuracy of 0.73, an AUC of 0.76-0.77, and an F1 score of 0.80, with the paper claiming this outperforms a GPT-4V baseline and existing supervised and MLLM-based methods. An ablation study over eight input configurations and a qualitative analysis of a few examples are also presented.

Significance. If the central claim held, this would be a notable demonstration that a video-based multimodal LLM can perform zero-shot pedestrian intention prediction competitively with or better than specialized supervised models, which is relevant for autonomous driving applications that require rapid adaptation to new scenarios. The paper's strengths include a clear task formulation, a consistent prompt design, the use of temporal video input rather than discrete frames, and an ablation study that isolates modality contributions. However, the quantitative claims are currently weakened by a post-hoc configuration selection on the test set, internal numerical inconsistencies, a mismatch between the reported input modalities and the best configuration, and the reliance on ground-truth bounding boxes in the annotated setting. These issues must be resolved before the significance of the result can be assessed reliably.

major comments (4)
  1. [Section III.F / Table II] The headline 0.73 accuracy is obtained by selecting the best-performing row (AV+S) among eight configurations, all evaluated on the JAAD test set, with no validation split or multiple-comparison correction. This post-hoc best-of-eight selection produces an optimistically biased estimate: the average accuracy across Table II is approximately 0.66, and the selected maximum will systematically overstate the performance of any fixed pipeline. Please pre-specify the configuration or use the JAAD validation set for selection, and report the test-set accuracy of the pre-specified model.
  2. [Abstract, Introduction, Table I] The claim of an '18% improvement' over GPT4V-PBP is numerically inconsistent with Table I: BF-PIP accuracy is 0.73 versus GPT4V-PBP's 0.57, which is an absolute difference of 0.16 (16 percentage points) and a relative improvement of 28%, not 18%. Additionally, Section III.D reports an AUC of 0.76 while Table I reports 0.77 for BF-PIP; these numbers should be reconciled.
  3. [Table I] The BF-PIP row in Table I lists B (bounding-box coordinates) and V (video) as separate inputs, but the best configuration AV+S from Table II uses annotated video (bounding boxes rendered onto frames) and ego-vehicle speed, without explicit bounding-box coordinate inputs. The ablation shows AV+BB+S (0.68) is worse than AV+S (0.73), so the table should either report the AV+S configuration as using V and S only, or clarify that B in Table I denotes rendered boxes within the video. As written, the table misrepresents the method's actual inputs.
  4. [Section III.A] The annotated mode relies on ground-truth bounding boxes from JAAD, which are not available in real-world deployment. No experiments with an object detector's outputs (including false positives, misses, or jitter) are reported, so the claimed 73% accuracy represents an upper bound under perfect localization. To support the claim of readiness for autonomous driving, please evaluate with a detector or state this limitation explicitly.
minor comments (5)
  1. [Affiliations] The word 'Departmrnt' should be 'Department' (it appears twice in the author affiliations).
  2. [Table I caption] The caption reads 'FROM OMNI PREDICT []' with an empty citation; it should reference [14].
  3. [Section II.C] The output constraint is described first as 'a single-word prediction' and later as 'structured JSON format'; please clarify the actual output format that was used and parsed.
  4. [Section II.C] The full prompt is not included in the paper; to make the 'consistent and reproducible' claim credible, please provide the exact prompt in an appendix or supplementary material.
  5. [Section III.C] Although the model is run five times per instance, no statistics about the variability of the five outputs are reported; please provide the per-instance agreement or variance.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported 73% accuracy is the post-hoc best-of-eight test-set configuration, making the zero-shot accuracy claim a fitted-result rather than a pre-specified prediction.

  1. fitted input called prediction [Section III.F Table II, Section III.D Table I, Abstract]
    "The 8 configurations were evaluated ... Results are measured using the five metrics mentioned previously and are summarized in Table II. ... The best performance is achieved with the A V + S configuration, reaching the highest accuracy of 0.73"

    The headline claim that BF-PIP achieves 73% accuracy is taken from the AV+S row of Table II, which was selected as the best of eight configurations, all evaluated on the JAAD test set. Table I then reports the same 0.73 as BF-PIP's performance. Because the configuration identity is chosen after inspecting test-set accuracies, the reported 0.73 is the maximum of the ablated scores—not an independent estimate for a pre-specified pipeline. The selection of the configuration is therefore a fitted parameter, and the reported accuracy is the selection statistic itself, which is statistically forced upward relative to a randomly pre-chosen configuration.

full rationale

The paper presents no formal derivation; BF-PIP is a Gemini 2.5 Pro prompting pipeline evaluated on the JAAD-beh benchmark. Per-instance outputs are genuine API predictions, and comparisons to external baselines are made on a public test set, so there is no self-citation or imported-uniqueness circularity. The only load-bearing circular step is the configuration selection: the paper evaluates eight input configurations on the JAAD test set, identifies AV+S as the best by accuracy, and then presents that same 0.73 as the zero-shot BF-PIP result. This is a fitted-input-called-prediction issue: the method identity is selected from the test labels, so the headline accuracy is the max of the ablated accuracies rather than an unbiased prediction for a predefined method. The '18% improvement' wording is also inconsistent with Table I (0.73 vs 0.57 is 16 percentage points, or 28% relative), but that is an arithmetic presentation error, not circularity. Aside from the test-set configuration selection, I find no other circular steps.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-chosen input modalities and an unreleased prompt, plus assumptions about benchmark comparability, ground-truth annotations, and model cleanliness. No new physical entities are postulated.

free parameters (2)
  • Input modality configuration = Annotated video + speed (AV+S), ACC 0.73
    Selected as the best of eight ablation configurations in Table II evaluated on the JAAD test set. The reported performance is the maximum over these choices, not a pre-specified pipeline.
  • Prompt design = Not disclosed
    The exact prompt text (role-play, chain-of-thought, output constraints) is described only qualitatively and not released. Wording choices can change MLLM accuracy and may have been iterated on the test set.
assumptions (4)
  • domain assumption JAAD-beh test protocol (16-frame observations, 30-frame TTE) is equivalent to the protocol used for the cited baselines.
    The paper compares with numbers from prior papers (Table I) under different observation lengths, e.g., GPT4V-PBP uses 10 frames, without re-running baselines.
  • domain assumption Ground-truth bounding boxes are available at inference time for the main annotated setting.
    Main result uses AV+S, which requires bounding boxes rendered from the dataset.
  • domain assumption Gemini 2.5 Pro's responses are not contaminated by prior exposure to JAAD test clips.
    Zero-shot claim assumes the model has not memorized the benchmark; this is unverified for a commercial model trained on web data.
  • ad hoc to paper The hand-crafted prompt is the only one used and is consistent across all runs.
    The prompt text is a paper-specific artifact that is not provided, making reproduction impossible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues." pith.science (2026). https://pith.science/paper/EACVMQQ5

@misc{pith2026250721161,
  author       = {Pith},
  title        = {Pith review of: Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EACVMQQ5}},
  note         = {Machine review of arXiv:2507.21161}
}
read the original abstract

Pedestrian intention prediction is essential for autonomous driving in complex urban environments. Conventional approaches depend on supervised learning over frame sequences and require extensive retraining to adapt to new scenarios. Here, we introduce BF-PIP (Beyond Frames Pedestrian Intention Prediction), a zero-shot approach built upon Gemini 2.5 Pro. It infers crossing intentions directly from short, continuous video clips enriched with structured JAAD metadata. In contrast to GPT-4V based methods that operate on discrete frames, BF-PIP processes uninterrupted temporal clips. It also incorporates bounding-box annotations and ego-vehicle speed via specialized multimodal prompts. Without any additional training, BF-PIP achieves 73% prediction accuracy, outperforming a GPT-4V baseline by 18 %. These findings illustrate that combining temporal video inputs with contextual cues enhances spatiotemporal perception and improves intent inference under ambiguous conditions. This approach paves the way for agile, retraining-free perception module in intelligent transportation system.

Figures

Figures reproduced from arXiv: 2507.21161 by the authors.

Figure 1
Figure 1. BF-PIP Framework In this work, we introduce BF-PIP, a zero-shot pedestrian intention prediction framework that leverages short video clips and along with structured metadata (bounding boxes and ego-vehicle speed) in a multimodal prompt to Gemini 2.5 Pro. In contrast to previous methods that relied on image sequences, our method allows for con￾tinuous motion capture, enabling spatiotemporal reasoning without the need… view at source ↗
Figure 2
Figure 2. Pedestrian crossing intention prediction. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 13 canonical work pages

  1. [1]

    Amir Rasouli, Iuliia Kotseruba, Toni Kunic, and John K. Tsotsos. Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction. In International Conference on Computer Vision (ICCV), 2019

    work page 2019

  2. [2]

    Do they want to cross? understanding pedestrian intention for behavior prediction

    Iuliia Kotseruba, Amir Rasouli, and John K Tsotsos. Do they want to cross? understanding pedestrian intention for behavior prediction. In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 1688–1693. IEEE, 2020

    work page 2020

  3. [3]

    Long-term on-board prediction of people in traffic scenes under uncertainty

    Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Long-term on-board prediction of people in traffic scenes under uncertainty. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4194–4202, 2018

    work page 2018

  4. [4]

    Pedestrian action anticipation using contextual feature fusion in stacked rnns

    Amir Rasouli, Iuliia Kotseruba, and John K Tsotsos. Pedestrian action anticipation using contextual feature fusion in stacked rnns. arXiv preprint arXiv:2005.06582, 2020

    arXiv 2005

  5. [5]

    Pedestrian graph +: A fast pedestrian crossing prediction model based on graph convolutional networks

    Pablo Rodrigo Gantier Cadena, Yeqiang Qian, Chunxiang Wang, and Ming Yang. Pedestrian graph +: A fast pedestrian crossing prediction model based on graph convolutional networks. IEEE Transactions on Intelligent Transportation Systems, 23(11):21050–21061, 2022

    work page 2022

  6. [6]

    St cross- ingpose: A spatial-temporal graph convolutional network for skeleton- based pedestrian crossing intention prediction

    Xingchen Zhang, Panagiotis Angeloudis, and Yiannis Demiris. St cross- ingpose: A spatial-temporal graph convolutional network for skeleton- based pedestrian crossing intention prediction. IEEE Transactions on Intelligent Transportation Systems, 23(11):20773–20782, 2022

    work page 2022

  7. [7]

    Pit: Progressive interaction transformer for pedestrian crossing intention prediction

    Yuchen Zhou, Guang Tan, Rui Zhong, Yaokun Li, and Chao Gou. Pit: Progressive interaction transformer for pedestrian crossing intention prediction. IEEE Transactions on Intelligent Transportation Systems, 2023

    work page 2023

  8. [8]

    Intformer: Predicting pedestrian intention with the aid of the transformer architecture

    Javier Lorenzo, Ignacio Parra, and MA Sotelo. Intformer: Predicting pedestrian intention with the aid of the transformer architecture. arXiv preprint arXiv:2105.08647, 2021

    arXiv 2021

Show all 19 references
  1. [9]

    Multi-input fusion for practical pedestrian intention prediction

    Ankur Singh and Upendra Suddamalla. Multi-input fusion for practical pedestrian intention prediction. In 2021 IEEE/CVF International Con- ference on Computer Vision Workshops (ICCVW), pages 2304–2311, 2021

    2021

  2. [10]

    Mcip: Multi-stream network for pedestrian crossing intention prediction

    Je-Seok Ham, Kangmin Bae, and Jinyoung Moon. Mcip: Multi-stream network for pedestrian crossing intention prediction. In European Con- ference on Computer Vision, pages 663–679. Springer, 2022

    2022

  3. [11]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  4. [12]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  5. [13]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  6. [14]

    Ham, Je-Seok, Jia Huang, Peng Jiang, Jinyoung Moon, Yongjin Kwon, Srikanth Saripalli, and Changick Kim. ”OmniPredict: GPT-4o Enhanced Multi-modal Pedestrian Crossing Intention Prediction.” In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learn- ing, Ne...

    2024

  7. [15]

    Gpt- 4v takes the wheel: Evaluating promise and challenges for pedestrian behavior prediction

    Jia Huang, Peng Jiang, Alvika Gautam, and Srikanth Saripalli. Gpt- 4v takes the wheel: Evaluating promise and challenges for pedestrian behavior prediction. arXiv preprint arXiv:2311.14786, 2023

    2023 arXiv

  8. [16]

    Predicting pedestrian crossing intention with feature fusion and spatio-temporal attention

    Dongfang Yang, Haolin Zhang, Ekim Yurtsever, Keith A Redmill, and ¨Umit ¨Ozg¨uner. Predicting pedestrian crossing intention with feature fusion and spatio-temporal attention. IEEE Transactions on Intelligent Vehicles, 7(2):221–230, 2022

    2022

  9. [17]

    Benchmark for evaluating pedestrian action prediction

    Iuliia Kotseruba, Amir Rasouli, and John K Tsotsos. Benchmark for evaluating pedestrian action prediction. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1258–1268, 2021

    2021

  10. [19]

    ”Role play with large language models.” Nature 623, no

    Shanahan, Murray, Kyle McDonell, and Laria Reynolds. ”Role play with large language models.” Nature 623, no. 7987 (2023): 493-498

    2023

  11. [20]

    ”Towards revealing the mystery behind chain of thought: a theoretical perspective.” Advances in Neural Information Processing Systems 36 (2023): 70757-70798

    Feng, Guhao, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. ”Towards revealing the mystery behind chain of thought: a theoretical perspective.” Advances in Neural Information Processing Systems 36 (2023): 70757-70798

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.