Predicting Road Crossing Behaviour using Pose Detection and Sequence Modelling

Agniva Roy; Jaydip Sen; Preetam Saha; Subhasis Dasgupta

REVIEW 3 major objections 5 minor 20 references

Predicting Road Crossing Behaviour using Pose Detection and Sequence Modelling

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pipeline that turns 15 frames of body-keypoint coordinates into a crossing-intent prediction finds GRU the most accurate sequence model, with 1D CNN fastest.

desk verdict The core claim is plausible but the current evaluation almost certainly leaks training and test data across the same video clips, so the reported AUCs don't yet establish generalization. read the letter →

arxiv 2508.15336 v1 pith:PQBNWM2N submitted 2025-08-21 cs.CV cs.AI

Subhasis Dasgupta , Preetam Saha , Agniva Roy , Jaydip Sen This is my paper · ORCID

classification cs.CVcs.AI

keywords pedestrianintentpredictionroadcrossingbehaviourposedetectionMediaPipeLSTMGRU1DCNNsequencemodelling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a computer can predict, from a short video of a person near a road, whether that person is about to cross. It builds an end-to-end pipeline that extracts 33 body keypoints per frame with MediaPipe, feeds 15-frame sequences of x-y coordinates into three sequence models, and labels the next frame as crossing or not. On a private dataset of 60 videos from 20 volunteers, the GRU achieved the best test AUC (89.24%), slightly above the LSTM (86.38%) and higher than the 1D CNN (74.27%), while the 1D CNN had the fastest inference (1 ms per window vs. 2–3 ms). The authors claim that pose landmarks alone carry enough signal to anticipate crossing intent, and that the speed/accuracy trade-off favors GRU for accuracy and CNN for speed. The same framework is proposed for autonomous vehicle decision-making and for other movement-intent sequence tasks.

What carries the argument

The central object is the sliding-window sequence of MediaPipe pose landmarks: 33 body keypoints per frame, reduced to their X and Y coordinates (66 values per frame), with a window of 15 frames used to classify the 16th frame as crossing or not. This representation converts raw video into a compact temporal signal. The comparison is carried by three sequence models—LSTM and GRU (two-layer, 50 hidden units, dropout 0.5) and a 1D CNN (kernel size 3, global average pooling)—all trained with binary cross-entropy and evaluated by accuracy and AUC on a common held-out test set.

What would settle it

Run the same MediaPipe + 15-frame-window pipeline with the three models on a public pedestrian-intention dataset (e.g., PIE) or have two independent annotators relabel the same 60 clips; if GRU's AUC no longer exceeds LSTM's on a larger sample, or the labelers disagree substantially, the ranking and the absolute AUC values in Table 1 would not transfer.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that the coordinates of a single pedestrian's body keypoints, extracted per frame by MediaPipe and read as 15-frame sequences, are sufficient for sequence models to infer whether the pedestrian is about to cross a road. The paper reports that GRU is the most accurate of the three models at this binary intent task on the held-out test set (AUC 89.24%), LSTM is close behind (86.38%), and 1D CNN is markedly lower (74.27%) but fastest (1 ms per window). The authors also claim that end-to-end processing including pose extraction runs at about 43 ms per frame on a consumer laptop, too slow for 30 fps video but usable at lower frame rates. The practica

Load-bearing premise

The load-bearing premise is that a human watching each frame can reliably label whether the pedestrian intends to cross, with no inter-annotator agreement reported, so the model scores inherit whatever noise or bias is in those manual labels.

Editorial extensions

If this is right

For this task, pose landmarks alone suffice: no background, object detector, or trajectory history is needed to get AUC above 86% for the recurrent models.
GRU gives the best accuracy/AUC trade-off, making it the candidate where precision matters more than milliseconds; 1D CNN is the candidate where speed matters most.
A 15-frame window (half a second at 30 fps) is enough to signal intent, which means the system can warn a vehicle about a likely crossing almost immediately.
The end-to-end 43 ms/frame latency means the current, unoptimised pipeline cannot keep up with 30 fps video, but it can run in real time on lower-frame-rate or downsampled feeds.
The same sequence-modelling setup can be transferred to other movement-intent tasks, such as sports analytics or detecting suspicious movement near restricted areas.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The recurrent models' edge over the 1D CNN may come from their ability to retain information across the 15-frame window; testing wider CNN kernels or dilated convolutions could close the gap.
The reported 43 ms per frame is dominated by MediaPipe and data handling rather than the sequence model, so optimizing pose extraction could make even the GRU reach 30 fps on similar hardware.
The manual labelling and lack of inter-annotator agreement mean a public-dataset replication would be the direct way to test whether GRU's ranking transfers to other annotators and scenes.
Scaling to multiple pedestrians in a frame, which the paper lists as future work, will require assigning landmarks to individuals and may degrade the clean single-person performance.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a small, honest, but methodologically under-powered study. The new bit is using MediaPipe 2D pose landmarks (66 coordinates per frame) as the sole input to LSTM/GRU/1D-CNN sequence models for pedestrian crossing intent. That specific input representation is a reasonable thing to try, and the paper is transparent about its setup: about 60 single-pedestrian clips, 20 volunteers, manual frame-level labeling, and a comparison of three standard sequence models. The authors also clearly state the single-pedestrian limitation and the speed/accuracy tradeoff. So there is a kernel of a useful empirical result here, if the evaluation holds up.

The problem is that the reported numbers (GRU AUC 89.24%, LSTM 86.38%, 1D-CNN 74.27%) are probably inflated by train/test leakage. The split is described as a random 90:10 split of 15-frame sliding windows. With only 60 videos and roughly 300 frames each, a random split at the window level almost certainly puts windows from the same video, same actor, and overlapping time steps into both train and test. Recurrent models can then memorize clip-specific pose dynamics, and the test set contains near-duplicates of training contexts. The paper never says the split was done video-wise or subject-wise, and the sentence about 'a few more videos were collected and kept aside to test the model' is never reconciled with the 90:10 split. That is a load-bearing weakness, not a cosmetic one.

The other soft spots are real but less severe: manual labels with no inter-annotator agreement, post-hoc reclassification of backtracking frames, no error bars or repeated splits, and no comparison on PIE or JAAD even though PIE is cited and discarded due to size. No code or data are released, so the numbers cannot be independently checked. These are all fixable, but as reported the central comparative claim about GRU versus LSTM is not established.

Still, this is not a paper to dismiss outright. The idea is simple, cheap, and potentially useful for low-resource autonomous driving scenarios. A serious referee could ask for a video-wise split, error bars, and a public benchmark, and the work could become a modest but valid empirical note. I'd send it to review if the authors are willing to address the leakage issue; otherwise I'd encourage them to revise first.

Referee Report

3 major / 5 minor

Summary. The paper proposes an end-to-end framework for predicting pedestrian road-crossing intent from video. Pose landmarks are extracted with MediaPipe (33 keypoints, X/Y coordinates), and three sequence models—LSTM, GRU, and 1D CNN—are trained on 15-frame windows to classify the next frame as crossing/not crossing. Experiments on a self-collected dataset of 60 videos (about 17,408 windows) report that GRU achieves the best test AUC (89.24%), followed by LSTM (86.38%) and 1D CNN (74.27%), with 1D CNN being fastest (1 ms inference). The authors claim the framework can support autonomous driving applications.

Significance. If the reported results were valid, the paper would offer a lightweight, landmark-based alternative to full-video models for pedestrian intent prediction, with potential real-time application. The use of MediaPipe for pose extraction and the comparison of three sequence models are reasonable design choices, and the authors are transparent about computational constraints. However, the evaluation methodology has fundamental weaknesses—most notably the lack of a video-wise train/test split and the reliance on subjective, retrospectively modified labels—that cast doubt on the quantitative claims. The paper does not release code or data, so the results are not independently checkable. The core idea is plausible, but the evidence presented does not currently establish the claimed superiority of GRU over LSTM or the generalizability of the models.

major comments (3)

[Section IV, Table 1 and data split] The reported test performance is likely inflated by train/test leakage. The dataset consists of 60 video clips, each roughly 300 frames, and the 15-frame windows are almost certainly overlapping. The paper states only that the dataset was split into trainset and testset with a 90:10 ratio (Section IV), giving 15,585 and 1,823 data points. Without a video-wise or subject-wise split, many windows from the same clip—and even the same actor—appear in both training and test sets. Recurrent and convolutional models can then memorize clip-specific pose dynamics, so the test AUC values in Table 1 do not measure generalization to new pedestrians or new crossing episodes. The sentence in Section III that 'a few more videos were collected and kept aside to test the model' is also inconsistent with the 90:10 split described in Section IV. The authors should rerun the experiments with grouping by vid
[Sections III and IV, labeling protocol] The ground-truth labels are subjective and may include future information. Section III states that labels were assigned manually by observing frames individually, and Section IV adds that frames depicting backtracking were reclassified from 'crossing' to 'not crossing.' This reclassification is done after the fact: a frame at time t is labeled 0 because the pedestrian later decides to backtrack, even though at time t the crossing intention may have been present. Thus the labels are retrospective, not predictive, and the model is trained to recognize a label that depends on future events beyond the 15-frame input window. No inter-annotator agreement is reported, and no protocol for prospective labeling is defined. The authors should define a labeling rule that uses only information up to the current frame, and report annotation reliability.
[Section IV, evaluation statistics] Table 1 reports single point estimates for accuracy, AUC, and inference time without any measure of variance. With only 60 clips and class imbalance, the difference between GRU AUC (89.24%) and LSTM AUC (86.38%) may not be statistically significant. The paper also reports only ROC AUC; given class imbalance, precision-recall AUC or a confusion matrix would be more informative. The authors should provide confidence intervals, multiple runs, or k-fold cross-validation with video-wise grouping to support the claim that GRU is 'better in predicting intent' than LSTM.

minor comments (5)

[Throughout] There are numerous typographical errors: 'fame' for frame, 'Long Short Tern Memory' for Long Short-Term Memory, 'Robicqet' for Robicquet, 'Fe-Fei' for Fei-Fei, 'Savarse' for Savarese, and 'SLTM' for LSTM in reference [10]. A careful proofread is needed.
[Section III, figures] The cross-references to figures are inconsistent. The text says 'The pose landmarks are shown in Figure 1' but Figure 1 is a training data flowchart and Figure 2 shows key points. The figure labels should be corrected.
[Section III, equation for 1D CNN] The convolution equation uses `𝑤𝑖 𝑥𝑡−𝑖` with the summation index going forward; typically the kernel index is reversed. This is notationally confusing and should be fixed.
[Section IV, inference time] The reported average inference time of 43 ms per frame includes MediaPipe and intermediate data handling, but the paper does not specify whether this is on CPU or GPU; the earlier inference times in Table 1 are presumably GPU. Clarify the hardware and the measurement conditions.
[Section II, literature review] The phrase 'the existing pretrained models stayed updated' is unclear. It would be helpful to state explicitly that MediaPipe was chosen because it is actively maintained by Google.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper is an empirical model comparison with no self-referential derivation, fitted-input-as-prediction, or load-bearing self-citation.

full rationale

This is a supervised empirical study, not a formal derivation. The claimed result that GRU predicts crossing intent better than LSTM while 1D CNN is fastest is obtained by training three sequence models on MediaPipe pose landmarks and comparing test-set AUC, accuracy, and inference time. No equation in the paper defines a prediction in terms of its own output, and no fitted parameter is renamed as a prediction. The ground-truth labels are human judgments made by watching frames, not derived from the models, so the training objective is not circular. The paper contains no self-citations to prior work by the same authors, and no uniqueness or ansatz is imported from the authors' own earlier papers. The use of MediaPipe as a pretrained feature extractor is an external tool, not a result derived in this paper. Potential methodological concerns such as manual labeling, class imbalance, hyperparameter tuning on validation data, and the unclear train/test split (random 90:10 versus 'kept aside' videos) are validity or generalization issues, not circularity. Since no specific circular step can be exhibited with a quote, the appropriate score is 0.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several domain assumptions: that pose landmarks encode crossing intent, that Z coordinates are unnecessary, that manual labels are valid, and that acted single-pedestrian videos represent real behavior. All are plausible but none is externally validated. The fitted values are mostly standard hyperparameters.

free parameters (7)

sequence_length = 15 frames
Chosen by hand to predict the 16th frame; sets the temporal context for all models.
hidden_units = 50
Selected via manual hyperparameter tuning for LSTM and GRU.
dropout_probability = 0.5
Manual hyperparameter choice to reduce overfitting.
cnn_kernel_size = 3
Manual choice for the single 1D convolution layer.
epochs = 10
Fixed training duration; best model saved by validation AUC.
train_test_split_ratio = 90:10
Chosen split ratio; no random seed or video-wise separation described.
crossing_threshold = 0.5
Probability above 50% considered strong crossing intent (Section II).

assumptions (5)

domain assumption Human body pose landmarks are sufficient to indicate crossing intent.
Stated in Section III: 'It was assumed that human poses could give an indication whether the person was about to cross the road or not.'
domain assumption Pedestrians cross along the shortest route, perpendicular to the road edge, so the Z coordinate can be ignored.
Section III: 'the study focused on the road crossing behaviour of pedestrians who were using the shortest route to cross the road... Z axis would not have had any significant impact.'
domain assumption Manual frame-by-frame labeling provides valid ground truth for intent.
Section III: labels assigned 'manually by observing the frames individually'; no inter-annotator agreement reported.
domain assumption Acted, single-pedestrian videos from 20 volunteers generalize to real crossing behavior.
Section I describes an 'experimental setup where participants were asked to simulate road-crossing behaviours'; Section V acknowledges single-pedestrian limitation.
standard math Standard LSTM, GRU, and 1D CNN equations and the MediaPipe pretrained model are correct.
Background equations in Section III; MediaPipe used as an off-the-shelf pretrained extractor.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Predicting Road Crossing Behaviour using Pose Detection and Sequence Modelling." pith.science (2026). https://pith.science/paper/PQBNWM2N

@misc{pith2026250815336,
  author       = {Pith},
  title        = {Pith review of: Predicting Road Crossing Behaviour using Pose Detection and Sequence Modelling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PQBNWM2N}},
  note         = {Machine review of arXiv:2508.15336}
}

read the original abstract

The world is constantly moving towards AI based systems and autonomous vehicles are now reality in different parts of the world. These vehicles require sensors and cameras to detect objects and maneuver according to that. It becomes important to for such vehicles to also predict from a distant if a person is about to cross a road or not. The current study focused on predicting the intent of crossing the road by pedestrians in an experimental setup. The study involved working with deep learning models to predict poses and sequence modelling for temporal predictions. The study analysed three different sequence modelling to understand the prediction behaviour and it was found out that GRU was better in predicting the intent compared to LSTM model but 1D CNN was the best model in terms of speed. The study involved video analysis, and the output of pose detection model was integrated later on to sequence modelling techniques for an end-to-end deep learning framework for predicting road crossing intents.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 19 canonical work pages

[1]

Predicting pedestrian road-crossing assertiveness for autonomous vehicle control,

F. Camara et al., “Predicting pedestrian road-crossing assertiveness for autonomous vehicle control,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2018, pp. 2098–2103

work page 2018
[2]

Do they want to cross? understanding pedestrian intention for behavior prediction,

I. Kotseruba, A. Rasouli, and J. K. Tsotsos, “Do they want to cross? understanding pedestrian intention for behavior prediction,” in 2020 IEEE Intelligent Vehicles Symposium (IV), IEEE, 2020, pp. 1688–1693

work page 2020
[3]

Prediction of pedestrian crossing intentions at intersections based on long short-term memory recurrent neural network,

S. Zhang, M. Abdel-Aty, J. Yuan, and P. Li, “Prediction of pedestrian crossing intentions at intersections based on long short-term memory recurrent neural network,” Transp Res Rec, vol. 2674, no. 4, pp. 57–65, 2020

work page 2020
[4]

Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,

A. Sherstinsky, “Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,” Physica D, vol. 404, p. 132306, 2020

work page 2020
[5]

Kumar and A

D. Kumar and A. Sinha, Yoga pose detection and classification using deep learning. LAP LAMBERT Academic Publishing London, 2020

work page 2020
[6]

Combining detection and tracking for human pose estimation in videos,

M. Wang, J. Tighe, and D. Modolo, “Combining detection and tracking for human pose estimation in videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11088–11096

work page 2020
[7]

img2pose: Face alignment and detection via 6dof, face pose estimation,

V. Albiero, X. Chen, X. Yin, G. Pang, and T. Hassner, “img2pose: Face alignment and detection via 6dof, face pose estimation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 7617–7627

work page 2021
[8]

Fusion of 3D LIDAR and camera data for object detection in autonomous vehicle applications,

X. Zhao, P. Sun, Z. Xu, H. Min, and H. Yu, “Fusion of 3D LIDAR and camera data for object detection in autonomous vehicle applications,” IEEE Sensors Journal, vol. 20, no. 9, pp. 4901-4913, 2020

work page 2020

Show all 20 references

[9]

Pedestrian trajectory prediction via spatial interaction Transformer network,

T. Su, Y. Meng, and Y. Xu, “Pedestrian trajectory prediction via spatial interaction Transformer network,” in 2021 IEEE Intelligent Vehicles Symposium Workshops (IV Workshops), pp. 154-159, 2021

2021
[10]

Social SLTM: Human trajectory prediction in crowded spaces,

A. Alahi, K. Goel, V. Ramanathan, A. Robicqet, L. Fe-Fei, and S. Savarse, “Social SLTM: Human trajectory prediction in crowded spaces,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961-971, 2016

2016
[11]

Pedestrian trajectory prediction in crowded environments using social attention graph neural networks,

M. Zhong, Y. Chang, Y. Dang, and K. Wang, “Pedestrian trajectory prediction in crowded environments using social attention graph neural networks,” Applied Sciences, vol. 14, no 20, art id: 9349, 2024

2024
[12]

Pedestrian simulation with reinforcement learning: A curriculum-based approach

G. Vizzari and T. Cecconello, “Pedestrian simulation with reinforcement learning: A curriculum-based approach.” Future Internet, vol. 15, no. 1, art. id: 12, 2023

2023
[13]

Deep learning for pedestrian behavior understanding,

S. Gazzeh and A. Douik, “Deep learning for pedestrian behavior understanding,” in 2022 6th International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), Sfax, Tunisia, pp. 1-5, 2022

2022
[14]

American sign language recognition for alphabets using MediaPipe and LSTM,

B. Sundar and T. Bagyammal, “American sign language recognition for alphabets using MediaPipe and LSTM,” Procedia Comput Sci, vol. 215, pp. 642–651, 2022

2022
[15]

Hand tracking in 3d space using mediapipe and pnp method for intuitive control of virtual globe,

V. Chunduru, M. Roy, and R. G. Chittawadigi, “Hand tracking in 3d space using mediapipe and pnp method for intuitive control of virtual globe,” in 2021 IEEE 9th Region 10 Humanitarian Technology Conference (R10-HTC), IEEE, 2021, pp. 1–6

2021
[16]

Suspicious human activity recognition using 2d pose estimation and convolutional neural network,

A. S. Dileep, S. S. Nabilah, S. Sreeju, K. Farhana, and S. Surumy, “Suspicious human activity recognition using 2d pose estimation and convolutional neural network,” in 2022 International Conference on Wireless Communications Signal Processing and Networking (WiSPNET), IEEE, 2...

2022
[17]

Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,

A. Rasouli, I. Kotseruba, T. Kunic, and J. K. Tsotsos, “Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6262–6271

2019
[18]

HMM-based architecture for face identification,

F. Samaria and S. Young, “HMM-based architecture for face identification,” Image Vis Comput, vol. 12, no. 8, pp. 537–543, 1994

1994
[19]

CRF learning with CNN features for image segmentation,

F. Liu, G. Lin, and C. Shen, “CRF learning with CNN features for image segmentation,” Pattern Recognit, vol. 48, no. 10, pp. 2983–2992, 2015

2015
[20]

Long Short-term Memory,

S. Hochreiter, “Long Short-term Memory,” Neural Computation MIT-Press, 1997

1997

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Predicting pedestrian road-crossing assertiveness for autonomous vehicle control,

F. Camara et al., “Predicting pedestrian road-crossing assertiveness for autonomous vehicle control,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2018, pp. 2098–2103

work page 2018

[2] [2]

Do they want to cross? understanding pedestrian intention for behavior prediction,

I. Kotseruba, A. Rasouli, and J. K. Tsotsos, “Do they want to cross? understanding pedestrian intention for behavior prediction,” in 2020 IEEE Intelligent Vehicles Symposium (IV), IEEE, 2020, pp. 1688–1693

work page 2020

[3] [3]

Prediction of pedestrian crossing intentions at intersections based on long short-term memory recurrent neural network,

S. Zhang, M. Abdel-Aty, J. Yuan, and P. Li, “Prediction of pedestrian crossing intentions at intersections based on long short-term memory recurrent neural network,” Transp Res Rec, vol. 2674, no. 4, pp. 57–65, 2020

work page 2020

[4] [4]

Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,

A. Sherstinsky, “Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,” Physica D, vol. 404, p. 132306, 2020

work page 2020

[5] [5]

Kumar and A

D. Kumar and A. Sinha, Yoga pose detection and classification using deep learning. LAP LAMBERT Academic Publishing London, 2020

work page 2020

[6] [6]

Combining detection and tracking for human pose estimation in videos,

M. Wang, J. Tighe, and D. Modolo, “Combining detection and tracking for human pose estimation in videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11088–11096

work page 2020

[7] [7]

img2pose: Face alignment and detection via 6dof, face pose estimation,

V. Albiero, X. Chen, X. Yin, G. Pang, and T. Hassner, “img2pose: Face alignment and detection via 6dof, face pose estimation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 7617–7627

work page 2021

[8] [8]

Fusion of 3D LIDAR and camera data for object detection in autonomous vehicle applications,

X. Zhao, P. Sun, Z. Xu, H. Min, and H. Yu, “Fusion of 3D LIDAR and camera data for object detection in autonomous vehicle applications,” IEEE Sensors Journal, vol. 20, no. 9, pp. 4901-4913, 2020

work page 2020