{"id":"5795ac46-ea95-43e7-ae9f-a212fda4520e","arxiv_id":"2507.21161","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Gemini 2.5 Pro, prompted with 16-frame video clips and ego-vehicle speed, predicts pedestrian crossing intent at 73% accuracy on JAAD-beh without finetuning.","lead":"This paper tests whether Google's Gemini 2.5 Pro can predict if a pedestrian will cross the road by watching short video clips with extra cues like car speed. On a standard driving dataset the model reached 73% accuracy without any training, a few points above earlier methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 73% is a post-hoc best-of-eight test-set selection, not the accuracy of a pre-specified method.","rationale":"The reader's verdict of CONDITIONAL is appropriate: the paper presents an interesting zero-shot video-based approach, but the central 73% claim is not yet trustworthy. The reader's stated weakest assumption was the use of ground-truth bounding boxes in annotated mode; that is a valid real-world deployment concern. However, the more load-bearing issue is the selection of the best out of eight test-set ablations as the final method, which directly biases the headline number upward. The reader did note this in the rationale, but not as the primary weakest assumption, hence partial agreement. The internal inconsistency between the claimed inputs (Table I) and the selected configuration (Table II) further undermines reproducibility. These concerns are addressable with a pre-registered validation protocol, so conditional acceptance—pending re-analysis with proper validation and consistent configuration reporting—remains the right verdict. If the re-analysis fails to reproduce at least comparable accuracy, rejection would be warranted, but the current evidence is insufficient to reject the underlying idea outright.","tokens_in":7355,"tokens_out":6204,"duration_ms":65644,"concrete_test":"Hold out a validation split (or use the JAAD validation set), evaluate all eight Table II configurations on that validation split, pre-register the best configuration, then evaluate that configuration once on the held-out JAAD test split. If the selected configuration's accuracy falls below 0.73, the headline result is inflated by test-set selection; also report the mean and standard deviation across the five repeated runs described in Section II.C.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is the 73% accuracy in Table I, but this number comes from Table II's ablation study: the AV+S configuration is the best-performing row among eight configurations, all evaluated on the JAAD test set. No validation split is used to select the configuration; the authors simply identify BF-PIP with the best row after seeing test-set results. This post-hoc selection over eight comparisons inflates the reported accuracy—the expected accuracy of a randomly chosen configuration is closer to the average of Table II (about 0.66), and the selected maximum will overestimate the true performance of any fixed pipeline. The issue is compounded by an internal inconsistency: Table I lists BF-PIP's inputs as B (bounding-box coordinates), S (speed), and V (video), but the selected best configuration AV+S omits explicit BB coordinate inputs, while the full AV+BB+S configuration scores only 0.68. Furthermore, the abstract and introduction claim an 18% improvement over GPT4V-PBP, yet Table I shows 0.73 versus 0.57, an absolute gain of 0.16 and relative gains of 28% (or 33% against the Skip variant), not 18%. These inconsistencies mean the headline result is not a reliable, reproducible estimate for a clearly specified method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BF-PIP, a zero-shot pedestrian crossing intention prediction method built on Gemini 2.5 Pro. The method takes short video clips from the JAAD-beh dataset, optionally enriched with rendered bounding boxes (annotated mode) or raw coordinate values, plus ego-vehicle speed, and uses a hand-crafted two-stage prompt to produce a binary cross/not-cross prediction. Evaluation on the JAAD-beh test set reports a best accuracy of 0.73, an AUC of 0.76-0.77, and an F1 score of 0.80, with the paper claiming this outperforms a GPT-4V baseline and existing supervised and MLLM-based methods. An ablation study over eight input configurations and a qualitative analysis of a few examples are also presented.","tokens_in":7558,"tokens_out":4904,"duration_ms":46559,"significance":"If the central claim held, this would be a notable demonstration that a video-based multimodal LLM can perform zero-shot pedestrian intention prediction competitively with or better than specialized supervised models, which is relevant for autonomous driving applications that require rapid adaptation to new scenarios. The paper's strengths include a clear task formulation, a consistent prompt design, the use of temporal video input rather than discrete frames, and an ablation study that isolates modality contributions. However, the quantitative claims are currently weakened by a post-hoc configuration selection on the test set, internal numerical inconsistencies, a mismatch between the reported input modalities and the best configuration, and the reliance on ground-truth bounding boxes in the annotated setting. These issues must be resolved before the significance of the result can be assessed reliably.","major_comments":[{"comment":"The headline 0.73 accuracy is obtained by selecting the best-performing row (AV+S) among eight configurations, all evaluated on the JAAD test set, with no validation split or multiple-comparison correction. This post-hoc best-of-eight selection produces an optimistically biased estimate: the average accuracy across Table II is approximately 0.66, and the selected maximum will systematically overstate the performance of any fixed pipeline. Please pre-specify the configuration or use the JAAD validation set for selection, and report the test-set accuracy of the pre-specified model.","section":"Section III.F / Table II"},{"comment":"The claim of an '18% improvement' over GPT4V-PBP is numerically inconsistent with Table I: BF-PIP accuracy is 0.73 versus GPT4V-PBP's 0.57, which is an absolute difference of 0.16 (16 percentage points) and a relative improvement of 28%, not 18%. Additionally, Section III.D reports an AUC of 0.76 while Table I reports 0.77 for BF-PIP; these numbers should be reconciled.","section":"Abstract, Introduction, Table I"},{"comment":"The BF-PIP row in Table I lists B (bounding-box coordinates) and V (video) as separate inputs, but the best configuration AV+S from Table II uses annotated video (bounding boxes rendered onto frames) and ego-vehicle speed, without explicit bounding-box coordinate inputs. The ablation shows AV+BB+S (0.68) is worse than AV+S (0.73), so the table should either report the AV+S configuration as using V and S only, or clarify that B in Table I denotes rendered boxes within the video. As written, the table misrepresents the method's actual inputs.","section":"Table I"},{"comment":"The annotated mode relies on ground-truth bounding boxes from JAAD, which are not available in real-world deployment. No experiments with an object detector's outputs (including false positives, misses, or jitter) are reported, so the claimed 73% accuracy represents an upper bound under perfect localization. To support the claim of readiness for autonomous driving, please evaluate with a detector or state this limitation explicitly.","section":"Section III.A"}],"minor_comments":[{"comment":"The word 'Departmrnt' should be 'Department' (it appears twice in the author affiliations).","section":"Affiliations"},{"comment":"The caption reads 'FROM OMNI PREDICT []' with an empty citation; it should reference [14].","section":"Table I caption"},{"comment":"The output constraint is described first as 'a single-word prediction' and later as 'structured JSON format'; please clarify the actual output format that was used and parsed.","section":"Section II.C"},{"comment":"The full prompt is not included in the paper; to make the 'consistent and reproducible' claim credible, please provide the exact prompt in an appendix or supplementary material.","section":"Section II.C"},{"comment":"Although the model is run five times per instance, no statistics about the variability of the five outputs are reported; please provide the per-instance agreement or variance.","section":"Section III.C"}],"recommendation":"major_revision","confidential_remarks":"The paper is currently a preprint with several presentation issues, including typos, incomplete references, and internal numerical inconsistencies. The main scientific concern is the post-hoc selection of the best configuration on the test set, which should be addressed in revision. The contribution over OmniPredict is incremental (video input instead of discrete frames, Gemini 2.5 Pro instead of GPT-4o), but the ablation design is reasonable and the central idea is worth considering after the methodological issues are fixed. The editor may also wish to verify the comparability of input lengths and evaluation protocols across the baselines in Table I."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the headline here is not the number they claim. The 73% accuracy comes from picking the best row in an eight-way ablation (AV+S) after seeing test-set results, with no validation split. And the abstract's 18% improvement over GPT4V-PBP should be about 28% relative, or 16 points absolute, so the claim as written is wrong.\n\nThat said, the paper is not a waste of trees. The video-input variant is a sensible extension of the frame-based MLLM work; feeding 16-frame raw clips with speed metadata to Gemini 2.5 Pro is a natural idea worth testing. The authors also did the work in a reasonably careful way: fixed temperature and seed, five runs per instance, and a clean task formulation on JAAD-beh. The ablation is informative, even if it is misused to produce the headline.\n\nSoft spots, in descending order. First, the post-hoc selection problem. Eight configurations evaluated on the test set, and the best one becomes 'BF-PIP.' That inflates expected performance and makes the zero-shot framing murkier. Second, an internal inconsistency in the method's identity: Table I lists BF-PIP's inputs as B, S, V, but the selected best configuration is AV+S, which drops BB coordinates, while the full AV+BB+S scores 0.68. Either the method is AV+S and the table is sloppy, or the method uses all three and the headline is cherry-picked. The authors need to clarify. Third, the 18% claim does not match their own table; that kind of number error makes a referee's job harder. Fourth, no code, prompt templates, or error bars. The temperature-0/seed-0 setup is good, but reporting five runs without variance is unhelpful. Finally, the annotated mode uses ground-truth boxes, so the real-world assumption of a reliable detector is untested. That is a major caveat, not a fatal flaw.\n\nThe core idea is plausible and the paper deserves serious peer review, but not acceptance in its current form. If the authors pre-register a single configuration, use a validation split for any selection, correct the arithmetic, and release prompts and code, this becomes a solid workshop-level contribution. I would bring it to a reading group only after those fixes. I would not cite it yet.","headline":"A post-hoc best-of-eight test-set selection and a mismatch in the reported improvement sink the 73% headline, but the video-input idea and the ablation study are worth a serious referee's time.","tokens_in":8131,"tokens_out":2435,"would_cite":false,"duration_ms":24133,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BF-PIP claims that zero-shot video prompts, combining raw temporal clips with speed and bounding-box cues, predict pedestrian crossing intention at 73% accuracy on JAAD-beh, beating a GPT-4V baseline by 18%.","keywords":["pedestrian intention prediction","zero-shot learning","multimodal large language models","video understanding","autonomous driving","Gemini 2.5 Pro","JAAD benchmark","prompt engineering"],"falsifier":"Run the same prompt pipeline on the same JAAD-beh test clips but replace the ground-truth rendered boxes with boxes from a real-time detector (e.g., YOLO) and measure accuracy; if the result falls toward the 0.65 unannotated-video level, the 0.73 annotated-mode result is not robust to detection error.","tokens_in":7115,"feed_emoji":"🚶","tokens_out":11232,"duration_ms":100153,"temperature":0.7,"pith_summary":"BF-PIP asks whether a general multimodal model can infer pedestrian crossing intent from short video clips without any task-specific training. The paper claims it can: on the JAAD-beh benchmark, a prompt built around a 16-frame raw video clip plus ego-vehicle speed lets Gemini 2.5 Pro reach 73% prediction accuracy, 18% above a GPT-4V frame-based baseline and 6% above the prior multimodal baseline OmniPredict. The point of caring is that autonomy pipelines normally need supervised retraining to handle new scenes; if this holds, intention prediction becomes a prompt-engineering problem rather than a model-training problem. The authors also find that raw unannotated video alone reaches 65% accuracy, so temporal continuity itself carries much of the signal.","feed_headline":"Zero-shot video prompt reads pedestrian intent at 73%","feed_subtitle":"No training, just 16 frames of video plus speed cues: Gemini 2.5 Pro beats frame-based GPT-4V on JAAD-beh.","key_machinery":"The load-bearing object is the BF-PIP multimodal prompt pipeline. It takes a 16-frame video clip (about 0.5 seconds at 30 FPS), optional rendered bounding-box overlays from JAAD, ego-vehicle speed, and a two-stage instruction prompt that assigns the model an observer role and requests a single-word binary decision with chain-of-thought reasoning. The pipeline runs deterministically (temperature 0, fixed seed) and makes five predictions per instance. What carries the argument is the contrast between this temporally continuous input and the still-frame inputs of prior MLLM baselines: motion, hesitation, and gaze shifts are visible to the model rather than reconstructed from sparse images.","core_discovery":"The central claim is that continuous temporal video, rather than discrete frames, is what lets a zero-shot multimodal large language model anticipate pedestrian crossing behavior. Concretely, BF-PIP feeds Gemini 2.5 Pro a 16-frame clip sampled 30 frames before the crossing event, optionally renders the JAAD ground-truth bounding box onto each frame, and adds the ego-vehicle's speed class in a structured prompt. In the annotated-video-plus-speed configuration the model reaches 0.73 accuracy, 0.80 F1, and 0.96 precision on JAAD-beh, outperforming a GPT-4V frame baseline by 18% and the prior MLLM method OmniPredict by 6%, as well as several supervised sequence models. The ablation attributes the gain to combining visual annotation with speed: adding explicit bounding-box coordinate text to video actually hurt accuracy, while speed consistently helped.","pith_inferences":["The reported 0.73 result is an upper bound for real deployments that detect rather than annotate pedestrians: replacing ground-truth JAAD boxes with detector outputs would likely land between the 0.73 annotated-video-plus-speed and 0.65 unannotated numbers.","The ablation's counterintuitive result that raw coordinate text hurts performance suggests the model benefits from visually grounded boxes but cannot consume them as numbers; a natural extension is testing spatial-language descriptions ('the pedestrian is on the left sidewalk near the crosswalk') instead of $(x,y,w,h)$ tuples.","Because the prompt is dataset-agnostic, the same video-plus-speed recipe could be tested on newer pedestrian datasets (e.g., PIE) or on cyclists and scooters, which would tell whether the temporal-continuity advantage transfers beyond JAAD's distribution."],"forward_implications":["If BF-PIP's 73% accuracy holds, pedestrian crossing prediction at a 1-second horizon can be done without training data or fine-tuning, using only a short video clip and speed metadata.","The reported advantage over the frame-based GPT-4V baseline assigns a concrete value to temporal continuity: replacing still frames with raw video changes decision quality on the same benchmark.","Ego-vehicle speed is a cheap, reliable signal; the ablation shows adding speed to annotated video raises accuracy from 0.64 to 0.73, so speed should be retained in future prompt designs.","Raw unannotated video alone reaches 0.65 accuracy, meaning even without bounding boxes the model captures a large share of the crossing signal; this is the deployment-relevant number if no detector is available."],"supporting_citations":[{"why":"Supplies the JAAD-beh clip subset, the ground-truth bounding boxes and speed metadata, and the 30-frame TTE protocol used to build every test clip.","marker":"[2]"},{"why":"Gives the OmniPredict baseline and the comparison table of prior JAAD-beh accuracies that BF-PIP's numbers are measured against.","marker":"[14]"},{"why":"Defines the frame-based GPT4V-PBP baseline that BF-PIP claims to outperform by 18%, the core evidence for temporal video over still frames.","marker":"[15]"},{"why":"Establishes the JAAD action-prediction benchmark protocol and the 16-frame observation window used in all evaluations.","marker":"[17]"},{"why":"Provides the role-play prompting technique that structures the two-stage instruction prompt.","marker":"[19]"},{"why":"Motivates the chain-of-thought reasoning steps that the prompt asks the model to perform before the binary decision.","marker":"[20]"}],"fun_headline_variants":["Clips beat frames for zero-shot pedestrian intent","Video + speed cues predict crossing at 73% without training","Gemini 2.5 Pro reads intent from raw temporal video","Continuous video outperforms frames for intent","Zero-shot intent at 73% from 16-frame clips"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 73% result is measured with JAAD's ground-truth bounding boxes rendered into the clips, so the method assumes a deployed system can supply accurate per-pedestrian boxes at prediction time; with imperfect detector boxes the reported accuracy would likely drop.","fun_headline_variants_meta":{"raw":{"variants":["Clips beat frames for zero-shot pedestrian intent","Video + speed cues predict crossing at 73% without training","Gemini 2.5 Pro reads intent from raw temporal video","Continuous video outperforms frames for intent","Zero-shot intent at 73% from 16-frame clips"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000565,"raw_usage":{"total_tokens":2657,"prompt_tokens":905,"completion_tokens":1752,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1673}},"tokens_in":521,"tokens_out":1752,"duration_ms":16368,"temperature":1.0,"reasoning_tokens":1673,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:02:22.714117+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same prompt pipeline on the same JAAD-beh test clips but replace the ground-truth rendered boxes with boxes from a real-time detector (e.g., YOLO) and measure accuracy; if the result falls toward the 0.65 unannotated-video level, the 0.73 annotated-mode result is not robust to detection error.","supporting_citations":[{"cited_title":"Do they want to cross? understanding pedestrian intention for behavior prediction","cited_arxiv_id":null,"evidence_quote":"Supplies the JAAD-beh clip subset, the ground-truth bounding boxes and speed metadata, and the 30-frame TTE protocol used to build every test clip."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the OmniPredict baseline and the comparison table of prior JAAD-beh accuracies that BF-PIP's numbers are measured against."},{"cited_title":"GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction","cited_arxiv_id":"2311.14786","evidence_quote":"Defines the frame-based GPT4V-PBP baseline that BF-PIP claims to outperform by 18%, the core evidence for temporal video over still frames."},{"cited_title":"”Role play with large language models.” Nature 623, no","cited_arxiv_id":null,"evidence_quote":"Provides the role-play prompting technique that structures the two-stage instruction prompt."},{"cited_title":"”Towards revealing the mystery behind chain of thought: a theoretical perspective.” Advances in Neural Information Processing Systems 36 (2023): 70757-70798","cited_arxiv_id":null,"evidence_quote":"Motivates the chain-of-thought reasoning steps that the prompt asks the model to perform before the binary decision."}],"review_version":2}