{"id":"cfbe80ef-965b-4a49-a836-5bd820b5c629","arxiv_id":"2411.13302","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Reason-enriched annotations and a cross-modal vision-language model improve pedestrian crossing-intent prediction over prior methods.","lead":"This paper introduces a new dataset and AI method for self-driving cars that predicts whether a pedestrian will cross the road and also explains why. The method improves prediction accuracy, suggesting that human-style reasons can make intent predictions more reliable and more trustworthy for drivers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cross-modal fusion in MINDREAD never consumes reasons as input: C = F·X^T uses a fixed label-embedding matrix, so the experiments do not actually test whether textual reasons improve intent prediction.","rationale":"The reader's weakest assumption centers on annotation bias, which is real, but I see a more fundamental architectural gap: MINDREAD does not feed reasons into the model at inference. The cross-modal representation is a bilinear product with a static label-embedding matrix, so the reported +4.2% could come from a richer learned classifier rather than from semantic explanations. A random-embedding control would settle this. I also agree with the reader that the Sec III annotation protocol, which conditions reasons on intent labels, weakens causal claims. Both issues are empirical and addressable; the dataset and multi-task results remain useful. I therefore retain a conditional verdict rather than rejecting outright, but the central claim needs to be either re-tested with reasons as actual input or explicitly reframed as an auxiliary-label effect.","tokens_in":13179,"tokens_out":5892,"duration_ms":76857,"concrete_test":"Retrain the full MINDREAD model from Table VIII replacing Sentence-BERT reason embeddings and the CSEA co-occurrence adjacency matrix with random fixed vectors and a random matrix of the same dimensions, keeping all other components, losses, and training setup identical. If intent accuracy and F1 remain at roughly 87.6% and 95.0%, the natural-language content of the reasons is not the source of the gain, and the claims about 'WHY helping WHAT' should be reframed as an auxiliary-label effect; if performance drops significantly, semantic reason embeddings are load-bearing.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"To support the central claim that human-understandable reasons improve intent estimation, textual reasons must enter the model as information. In Sec IV-C the final cross-modal representation is C = F · X^T, where F is the visual attention output and X is the learned correlated semantic explanation embedding matrix from CSEA. Critically, X is a function only of the fixed set of reason categories and their co-occurrence statistics; it is not conditioned on the input video or on any per-pedestrian reason observation. Thus at both training and inference the reason 'modality' is a constant label-embedding matrix, and the model receives no text input. The gain in Table VIII therefore demonstrates that adding a parameterized reason-label embedding pathway helps, not that 'WHY' as semantic content helps 'WHAT'. The Sec III annotation protocol, which gives annotators the ground-truth intent before asking for reasons, compounds this: the reason labels may simply re-encode the intent label. Under both issues, the headline claim is untested as stated; the reported improvements are consistent with an auxiliary multi-task label-embedding effect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PIE++, an extension of the PIE dataset with multi-label textual reasons for pedestrian crossing/no-crossing intent, and proposes MINDREAD, a multi-task framework that fuses visual spatiotemporal features with learned reason embeddings via an attention mechanism. The authors report significant improvements over prior methods on PIE++ and JAAD, and provide ablations, user studies, and runtime analysis. The paper's central hypothesis is that predicting the \"WHY\" can improve prediction of the \"WHAT,\" i.e., that human-understandable reasons improve pedestrian intent estimation.","tokens_in":13420,"tokens_out":6012,"duration_ms":59237,"significance":"If the central claim is sustained, this is a novel and practically relevant contribution: it would be the first to use textual reasons to improve pedestrian intent prediction while also providing explainable outputs. The dataset (PIE++) is a potentially useful resource, and the user study adds evidence for the utility of reasons. However, the current experimental design does not actually test the claim because the reason modality is never used as input to the model; the reported gains are consistent with a multi-task label-embedding effect rather than with the semantic content of reasons. The annotation protocol also introduces a circularity risk. These issues must be addressed before the paper can be accepted.","major_comments":[{"comment":"The cross-modal representation is defined as C = F · X^T, where F is the visual attention output and X is the learned correlated semantic explanation embedding matrix from CSEA. X is a function only of the fixed set of reason categories and their co-occurrence statistics; it is not conditioned on the input video or on any per-pedestrian reason observation. Therefore, at both training and inference, the model receives no textual reason as input. The improvement in Table VIII (83.4% vs 87.6% intent accuracy) demonstrates that adding a parameterized reason-label embedding pathway helps, but it does not test whether human-understandable reasons improve intent prediction. To support the central claim, the model must ingest reasons as information (e.g., by conditioning on ground-truth or predicted per-instance reason labels), or the claims must be reframed to a multi-task label-embedding study.","section":"Sec. IV-C, Eq. (C = F · X^T)"},{"comment":"Annotators were provided with the ground-truth pedestrian intent annotations before selecting reasons. This makes the reason labels partly derived from the intent labels, so the reasons are not an independent signal for intent. The paper's hypothesis is that reasons help predict intent; if reasons are produced by annotators who already know the intent, the reported gains may be an artifact of the reasons re-encoding the intent labels. Please either collect reasons without revealing intent (or with a separate set of annotators), or provide an analysis showing that reasons carry information beyond the intent label (e.g., measure the mutual information between reasons and intent, or train the reason head on a subset without intent supervision).","section":"Sec. III, 'Sourcing Reason Annotations in PIE++'"},{"comment":"MINDREAD uses a Swin-V2-L + Transformer backbone, while the baseline methods (PIEint, STIP, CIA) use VGG + GRU. Table XI shows that replacing VGG + GRU with Swin-V2 + Transformer in MINDREAD improves intent accuracy by 0.6% (87.0% vs 87.6%). Since the reported improvement over CIA is 5.6%, the backbone difference is a confound in the SOTA comparison. Please provide results where all methods use the same backbone, or at least quantify the contribution of the backbone to the reported gains.","section":"Table V, SOTA comparison"}],"minor_comments":[{"comment":"The user study is based on only 5 subjects; please report the raw counts and discuss the limitations of this small sample size.","section":"Table IV"},{"comment":"The claim that this is the \"first pedestrian reason+intent prediction dataset\" is not fully substantiated; the PSI dataset [28] is cited but not compared as a reason-bearing dataset. Please clarify the distinction or discuss PSI explicitly.","section":"Sec. I"},{"comment":"The reason prediction accuracy metric is not defined; please specify whether it is exact-match multi-label accuracy, Hamming accuracy, or another metric, and state how it is computed.","section":"Table VI"},{"comment":"The name of this ablation suggests that the variant uses no text modality, but it still trains a reason classification head; please clarify that this ablation removes the CSEA module and the cross-modal fusion, rather than removing all text-related processing.","section":"Sec. VI, 'MINDREAD without Cross-modality'"},{"comment":"The adjacency matrix A is computed from co-occurrence of reasons \"within the dataset\"; please clarify whether these statistics are computed on the training split only to avoid information leakage through the validation/test splits.","section":"Sec. IV-A"},{"comment":"Please provide the dimensions of the tensors in the cross-modal fusion (F, X, and C) to make the operation concrete and reproducible.","section":"Sec. IV-C"}],"recommendation":"major_revision","confidential_remarks":"The paper introduces a valuable dataset and a clear multi-task formulation, and the user study is a nice addition. However, the central claim that reasons help improve intent prediction is not actually tested by the current architecture, since the reason modality is a constant label-embedding matrix. This is a load-bearing gap that requires a substantial experimental redesign or a significant reframing of the claims. The annotation circularity adds a further layer of concern. I encourage the authors to either modify the method to consume per-instance reason text (e.g., in a two-stage or late-fusion setting) or to reposition the paper as a study of multi-task label-embedding benefits. The paper fits the journal's scope but needs major revision before it can be considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper delivers a genuinely new resource—PIE++, with multi-label textual reasons for pedestrian intent—but the headline claim is not supported by the experiments as designed. The 'cross-modal' fusion never actually reads the reasons; it uses a fixed embedding of the reason label set. So the 5.6% gain is better described as an auxiliary label-embedding effect than as evidence that human-understandable 'WHY' helps 'WHAT'.\n\nWhat's good: the annotation effort is real. 4950 multi-label annotations on PIE, ICC 0.98, a user study on trust/usefulness, and the reason set is sensible. The multi-task setup with a reason head is a reasonable idea, and Table VIII is a clean ablation: removing the cross-modal pathway drops intent accuracy by 4.2%, so the extra pathway is doing something.\n\nThe trouble is in Sec IV-C. The cross-modal representation is C = F·X^T, where X is a learned matrix over the 17 reason categories and their co-occurrence statistics. It is not conditioned on any per-pedestrian reason, at either training or inference. No text input ever reaches the intent head. That means the experiment does not test whether textual reasons improve intent; it tests whether adding a parameterized reason-label embedding to a visual model helps. The annotation protocol compounds this: annotators were given the ground-truth intent before selecting reasons, and with only 17 categories the reasons may simply re-encode the intent. So the dataset is useful for benchmarking joint prediction, but it cannot support the causal claim in the title.\n\nMinor issues: the SOTA comparisons are not same-backbone, though Table XI shows the backbone accounts for only ~0.6% on PIE++, so that's not the main problem. JAAD results lack error bars, and no code/data release makes independent verification impossible.\n\nBottom line: a serious referee should engage with this—the dataset alone earns a look—but the paper needs a major rewrite. The authors should either reframe the claim as multi-task learning with shared label embeddings, or actually feed predicted reasons into the intent head and compare against a no-reason baseline. With that, it could be a solid contribution.","headline":"Useful new reason-annotated dataset, but the central claim that reasons improve intent is untested because the reason modality never enters the model as input.","tokens_in":13953,"tokens_out":3676,"would_cite":false,"duration_ms":38890,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding human-readable textual reasons to a vision pipeline improves pedestrian intent prediction, reporting 87.6% accuracy on PIE++ and 95.4% on JAAD.","keywords":["pedestrian intent prediction","cross-modal representation learning","textual reasons","PIE++ dataset","multi-task learning","autonomous driving","explainability"],"falsifier":"A concrete test is to collect reason annotations without revealing the crossing label to annotators and then retrain MINDREAD on that version. If intent accuracy no longer improves over the vision-only baseline (or the gain falls to the level of a model trained with a random auxiliary head), the paper's central claim would be refuted; the same test could be simulated by scrambling the reason-to-intent association in PIE++ and checking whether the 4.2% cross-modality gain persists.","tokens_in":12999,"feed_emoji":"🚶","tokens_out":4826,"duration_ms":44811,"temperature":0.7,"pith_summary":"This paper sets out to show that knowing why a pedestrian might cross helps predict whether they will cross. The authors build PIE++, a reason-enriched version of the PIE dataset with multi-label textual explanations for each pedestrian's intent, and propose MINDREAD, a multi-task model that fuses video features with learned embeddings of these reasons. Their experiments report that MINDREAD improves intent accuracy by 5.6% and F1 by 7% over the previous state of the art on PIE++, reaching 87.6% accuracy, and improves accuracy by 4.4% on JAAD. If the claim holds, both predictive accuracy and human interpretability of autonomous-driving systems improve, because the model can state a reason alongside its crossing decision.","feed_headline":"Learning why a pedestrian crosses lifts intent accuracy 5.6%","feed_subtitle":"MINDREAD fuses video with textual reasons, beating vision-only baselines on PIE++ and JAAD.","key_machinery":"The load-bearing component is the Correlated Semantic Explanation Affinity (CSEA) module, which treats the textual reasons as nodes in a directed graph whose edges are co-occurrence probabilities, then applies a graph convolutional network to produce correlated semantic embeddings. A transformer-based feature encoder (TFE) extracts spatial and temporal visual features from local context, global context, and bounding boxes, and an attention mechanism fuses the two representations into a shared cross-modal vector. This fused vector feeds two prediction heads, one for intent and one for multi-label reasons. The paper's hypothesis is that reason co-occurrence structure, captured by the graph, is what lets the language modality contribute beyond simple label information.","core_discovery":"The central claim is that the reason behind a pedestrian's intent is not just an explainability add-on but a useful learning signal: a cross-modal representation that joins spatiotemporal visual features with textual reason embeddings predicts the binary cross/no-cross intent better than vision-only baselines. On the PIE++ dataset the proposed MINDREAD model attains 87.6% accuracy and 95.0% F1, compared with 82.0% and 88.0% for the prior CIA method, and the reason-prediction head itself reaches 72.4% accuracy. Ablating the cross-modal fusion drops intent accuracy by 4.2%, which the authors take as evidence that the textual reasons carry information not present in the visual stream. On JAAD, enriched with the same reason categories, MINDREAD reaches 95.4% intent accuracy versus 91.0% for PV-LSTM.","pith_inferences":["A decisive test of the causal claim would re-collect reasons without showing annotators the ground-truth crossing label; if the accuracy gain shrinks, the current improvements may largely reflect an indirect encoding of the intent label through the multi-task head.","If reasons are truly informative, the same cross-modal recipe could be extended to other vulnerable road users (cyclists, motorcyclists) and to finer-grained intent distinctions such as crossing start time, which the paper does not test.","The reason taxonomy itself could be made richer (gaze direction, phone use, vehicle approach speed) and the affinity graph would then need to be re-learned; the paper's current graph is tied to its 17 reason categories.","Because the model outputs reasons, one could audit failures by checking whether the predicted reason is consistent with the predicted intent; such consistency checks are not reported in the paper."],"forward_implications":["Systems that predict pedestrian intent can output a human-readable reason alongside the binary decision, without sacrificing accuracy.","On the benchmarks used, the gains are consistent across both PIE++ and JAAD, suggesting the approach transfers across datasets.","The PIE++ annotations provide a resource for training and evaluating reason-based intent models.","Training a model to predict reasons jointly with intent yields better intent prediction than training intent alone, per Table VIII.","The method's runtime is comparable to or faster than a prior baseline (4.3 ms vs 4.7 ms on JAAD), so the cross-modal fusion does not obviously cost latency."],"supporting_citations":[{"why":"Supplies the base PIE dataset and the PIEint baseline that PIE++ enriches and that MINDREAD is compared against.","marker":"[1]"},{"why":"Provides the CIA method, the state-of-the-art baseline whose 82.0% intent accuracy and 88.0% F1 MINDREAD surpasses on PIE++.","marker":"[2]"},{"why":"Provides the STIP baseline for spatiotemporal relationship reasoning, another comparative method on PIE++.","marker":"[9]"},{"why":"Supplies the JAAD dataset used for the generalization experiment and the PV-LSTM comparison.","marker":"[16]"},{"why":"Sentence-BERT generates the sentence-level textual embeddings for each reason, the input to the CSEA graph.","marker":"[39]"},{"why":"Swin Transformer V2 is the visual backbone used to extract local and global spatiotemporal features in the TFE module.","marker":"[38]"},{"why":"Graph convolutional networks are the mechanism in CSEA that transform textual-reason embeddings into correlated semantic embeddings.","marker":"[44]"},{"why":"PV-LSTM is the baseline on JAAD that MINDREAD outperforms in intent accuracy and runtime.","marker":"[50]"}],"fun_headline_variants":["Why pedestrians cross: AI learns reasons to boost intent accuracy","Asking 'why' helps AI predict pedestrian crossings, 5.6% better","MINDREAD: reasoning improves pedestrian intent prediction by 5.6%","Cross-modal reasons lift pedestrian intent accuracy 5.6%","Pedestrian intent: adding 'why' boosts accuracy 5.6% on PIE++"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest premise is that the PIE++ reason annotations are independent signals for intent, even though annotators were shown the ground-truth crossing label before choosing reasons; if the reasons merely restate the intent label, the reported gains could come from the multi-task label embedding rather than from human-understandable reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Why pedestrians cross: AI learns reasons to boost intent accuracy","Asking 'why' helps AI predict pedestrian crossings, 5.6% better","MINDREAD: reasoning improves pedestrian intent prediction by 5.6%","Cross-modal reasons lift pedestrian intent accuracy 5.6%","Pedestrian intent: adding 'why' boosts accuracy 5.6% on PIE++"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00046,"raw_usage":{"total_tokens":2310,"prompt_tokens":960,"completion_tokens":1350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":1250}},"tokens_in":576,"tokens_out":1350,"duration_ms":11737,"temperature":1.0,"reasoning_tokens":1250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:35:02.344540+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test is to collect reason annotations without revealing the crossing label to annotators and then retrain MINDREAD on that version. If intent accuracy no longer improves over the vision-only baseline (or the gain falls to the level of a model trained with a random auxiliary head), the paper's central claim would be refuted; the same test could be simulated by scrambling the reason-to-intent association in PIE++ and checking whether the 4.2% cross-modality gain persists.","supporting_citations":[{"cited_title":"Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,","cited_arxiv_id":null,"evidence_quote":"Supplies the base PIE dataset and the PIEint baseline that PIE++ enriches and that MINDREAD is compared against."},{"cited_title":"Cou- pling intent and action for pedestrian crossing behavior prediction,","cited_arxiv_id":null,"evidence_quote":"Provides the CIA method, the state-of-the-art baseline whose 82.0% intent accuracy and 88.0% F1 MINDREAD surpasses on PIE++."},{"cited_title":"Spatiotemporal relationship reasoning for pedestrian intent prediction,","cited_arxiv_id":null,"evidence_quote":"Provides the STIP baseline for spatiotemporal relationship reasoning, another comparative method on PIE++."},{"cited_title":"Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior,","cited_arxiv_id":null,"evidence_quote":"Supplies the JAAD dataset used for the generalization experiment and the PV-LSTM comparison."},{"cited_title":"Sentence-bert: Sentence embeddings using siamese bert-networks,","cited_arxiv_id":null,"evidence_quote":"Sentence-BERT generates the sentence-level textual embeddings for each reason, the input to the CSEA graph."},{"cited_title":"Swin transformer v2: Scaling up capacity and resolution,","cited_arxiv_id":null,"evidence_quote":"Swin Transformer V2 is the visual backbone used to extract local and global spatiotemporal features in the TFE module."},{"cited_title":"Semi-supervised classification with graph convolutional networks,","cited_arxiv_id":null,"evidence_quote":"Graph convolutional networks are the mechanism in CSEA that transform textual-reason embeddings into correlated semantic embeddings."}],"review_version":1}