Multi-timescale Trajectory Prediction for Abnormal Human Activity Detection

Neha Bhargava; Rajbabu Velmurugan; Royston Rodrigues; Subhasis Chaudhuri

REVIEW 2 major objections 4 minor 18 references

Multi-timescale Trajectory Prediction for Abnormal Human Activity Detection

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that detecting abnormal human activities requires predicting pose trajectories at several timescales at once, and that a multi-layered 1D convolutional network supervised at timescales 3, 5, 13, and 25 outperforms…

desk verdict A clean multi-timescale extension of pose-based anomaly detection with a genuinely useful new dataset, but the headline gain rests on a hand-picked uniform timescale average and single-run numbers. read the letter →

arxiv 1908.04321 v1 pith:4ZTNNQHK submitted 2019-08-12 cs.CV

Royston Rodrigues , Neha Bhargava , Rajbabu Velmurugan , Subhasis Chaudhuri This is my paper · ORCID

classification cs.CV

keywords abnormalactivitydetectiontrajectorypredictionmulti-timescaleposetrajectoriessurveillancevideoanomalyconvolutionalneuralnetworkCorridordataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that abnormal human activities have no single temporal scale: a jump is a short perturbation, while loitering becomes abnormal only after many frames. A detector trained at one fixed timescale will therefore miss anomalies whose duration it never saw. To fix this, the authors build a model that predicts past and future pose trajectories at four timescales simultaneously, with intermediate network layers supervised to predict 3, 5, 13, and 25 steps ahead. The prediction errors from all timescales are averaged into an anomaly score per frame. A sympathetic reader would take away that multi-timescale supervision is a simple, label-free way to make video anomaly detection cover both brief and prolonged human behaviors.

What carries the argument

The load-bearing mechanism is a multi-layered 1D convolutional encoder-decoder in which intermediate layers are supervised, through node-level and layer-level losses, to predict trajectory offsets at chosen time horizons. Layer 1 has a receptive field of 3 frames and predicts 3 steps ahead; layers 2, 4, and 7 correspond to horizons 5, 13, and 25. A sliding window over the input produces several predictions per time instant, and these are averaged into layer losses, so training mimics the sliding-window test procedure. Past predictions come from running the same architecture on the reversed trajectory. The final anomaly score at time $t$ is the equal-weight average over timescales of the per-layer losses, taking the maximum over all people in the frame.

What would settle it

A single experiment would settle it: train the same architecture with only the 25-step supervision and only the 3-step supervision, and compare both against the combined model on the Corridor dataset's loitering and sudden-running sequences. If the combined model does not beat the best single timescale on both anomaly types, the multi-timescale claim fails.

Watch

Extended reading notes

Core claim

The central claim is that prediction error signals abnormality only at the right timescale, so a model should predict at many timescales and combine the errors. The authors train a seven-layer 1D convolutional network on human pose trajectories, supervising layers 1, 2, 4, and 7 to predict 3, 5, 13, and 25 steps into the future, and a twin model that makes the same predictions into the past. At test time, per-timescale errors are averaged to score each frame, and frames above a threshold are tagged abnormal. On HR-ShanghaiTech the full model reaches 77.04 Frame-AUC and on HR-Avenue 88.33, exceeding the compared single-timescale baselines; the ablation shows AUC climbing from 72.05 with only 3-step prediction to 77.04 with all four timescales. The paper also introduces the Corridor dataset, 483,566 annotated frames of staged single- and group-level anomalies.

Load-bearing premise

The central claim rests on the assumption that the four hand-picked timescales {3, 5, 13, 25} are the right ones and that averaging their prediction errors with equal weight is a good anomaly score.

Editorial extensions

If this is right

A fixed-horizon predictor can be upgraded to a multi-timescale one by adding supervision at intermediate layers, without any new labels.
On HR-ShanghaiTech, Frame-AUC rises monotonically as timescales 3, 5, 13, and 25 are added, from 72.05 to 77.04, so long-horizon errors carry information that short-horizon errors miss.
Combining past and future predictions improves over either direction alone in most tested configurations.
Pose trajectories alone are sufficient to detect human anomalies; the max-over-people aggregation makes the method applicable to group scenes.
The chosen timescales must match the anomalies present: on HR-Avenue, adding the 25-step timescale slightly reduces AUC because that dataset lacks long-term anomalies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The equal-weight voting rule is the most brittle part of the design; a learned weighting of the four timescales would probably improve datasets with mixed anomaly durations, since Table III already shows the 25-step timescale hurting HR-Avenue.
The timescale set {3, 5, 13, 25} is fixed by the network's receptive fields rather than by anomaly statistics; selecting horizons from the training data could reveal whether an intermediate duration such as 9 steps matters.
The same intermediate-supervision trick could be applied to other sequential abnormality signals, such as optical flow or object trajectories, not only human skeletons.
The paper reports only Frame-AUC, so per-anomaly recall by duration is untested; that measurement would directly validate or undermine the multi-timescale explanation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper does something straightforward and useful: it takes Morais et al.'s pose-trajectory prediction approach and gets a single 1D convolutional stack to predict at several timescales by supervising intermediate layers whose receptive fields are 3, 5, 13, and 25 steps. That is a real architectural idea, and the ablations in Table III do show that adding timescales helps on HR-ShanghaiTech (72.05 to 77.04 AUC for Future+Past). The new Corridor dataset is the other solid contribution: 483k frames, larger than the usual single-camera benchmarks, and it includes group-level anomalies (protest, chasing, fighting) that other datasets lack. I would want the dataset released, but the scale and activity variety are valuable on their own.

The soft spots are about robustness, not about the core idea. The stress-test note is on target: Eq. (7) averages prediction errors uniformly over the fixed, hand-picked set {3,5,13,25}. Table III is the only evidence for the multi-timescale benefit, and it is not monotonic—on HR-Avenue, adding the 25-step timescale drops AUC from 88.43 to 88.33. The authors blame the absence of long-term anomalies in Avenue, but they provide no per-timescale or per-anomaly analysis to verify that explanation. They also appear to have selected the final configuration after looking at test-set numbers, with no validation split, no error bars, and no significance tests. All results are single runs. The abstract's claim that the model "outperforms existing methods" is too strong: on the full Avenue benchmark they are at 82.85, below Liu et al.'s 84.90. The weaknesses are addressable—error bars, a validation-based timescale choice or learned weights, a softened abstract, and released code and data—but as written the paper's central claim depends on choices that could reasonably shift the outcome.

What is genuinely good: the intermediate-supervision mechanism is elegant, the loss design with node and layer terms is sensible, and the paper is clearly written. The citation pattern looks fine; the work builds directly on Morais et al. and says so. I would not desk-reject this. It deserves a serious referee and probably a conditional accept after experimental rigor is tightened. The multi-timescale principle is plausible, and the dataset alone justifies engaging with the paper.

For peer review: send it out, but the reviewers should push on the timescale selection, the uniform weighting, and the reproducibility of the numbers.

Referee Report

2 major / 4 minor

Summary. This paper proposes a multi-timescale framework for abnormal human activity detection in videos. The input is a human pose trajectory; two 1D-convolutional models predict future and past trajectories at four timescales (3, 5, 13, and 25 steps), with supervision at intermediate layers whose receptive fields match these horizons. At test time, prediction errors from all timescales and both directions are averaged (Eq. 7, with max over persons in Eq. 8) and thresholded to tag abnormal frames. The authors introduce a new Corridor dataset with 483,566 annotated frames and evaluate on HR-ShanghaiTech, HR-Avenue, Avenue, and ShanghaiTech. Table I reports improvements over prior methods on the HR variants, and Table III ablates the effect of adding timescales and the past-prediction branch.

Significance. If the results are reliable, the paper makes a useful contribution by showing that supervising intermediate convolutional layers at multiple temporal horizons yields prediction errors that are more discriminative for anomaly detection than a single timescale. The Corridor dataset is a substantial new resource with multi-person anomalies and class labels. The ablations in Table III provide direct evidence for the multi-timescale principle, and the architecture and loss are described in sufficient detail to be reimplemented. The main weaknesses are the lack of statistical validation and the hand-picked, uniformly weighted timescale set, which make the quantitative claims less robust than the text suggests.

major comments (2)

[V-E (Table III) and Eq. (7)] The central claim rests on the specific timescale set S = {3, 5, 13, 25} and the uniform averaging in Eq. (7). Table III shows that adding the 25-step timescale reduces HR-Avenue AUC from 88.43 to 88.33, which the authors attribute to the dataset lacking long-term anomalies, but no per-anomaly or per-timescale analysis is provided to test that explanation. Because the final configuration appears to be selected after observing the test-set results, the reported gains could be configuration-specific. Please provide validation-based selection of S, a sensitivity analysis over S and the weights in Eq. (7), or a per-anomaly breakdown demonstrating that different timescales contribute to different anomaly types.
[V-D (Tables I and III)] All reported Frame-AUC values are point estimates from a single run. The margins between configurations are small (e.g., 75.65 vs 77.04 on HR-ShanghaiTech; 88.43 vs 88.33 on HR-Avenue) and may be within run-to-run variance for a deep model trained with this protocol. Please report the mean and standard deviation over multiple training runs, or confidence intervals via bootstrap over the test set, and state whether the differences versus the baselines and ablations are statistically significant.

minor comments (4)

[Abstract and Section V-D] The abstract claims the model 'outperforms existing methods' without qualification, but Table I shows lower Frame-AUC on the full Avenue dataset (82.85) than Liu et al. (84.90). The body qualifies the claim to HR-ShanghaiTech and HR-Avenue; the abstract should be corrected to avoid overclaim.
[III-C (Eq. 3)] The definition of L_2^j in Eq. (3) is inconsistent with Eqs. (1) and (7): as written it defines a per-time average error, yet Eq. (1) uses it as a sum over t and Eq. (7) uses L_2^j(t). Please clarify the notation, e.g., define L_2^j(t) explicitly and write the layer-level loss as a sum over t.
[V-B] The sentence 'The loss after an epoch is equal to the loss incurred at the last sub-epoch' is ambiguous about whether gradients from earlier sub-epochs are retained. Please specify the optimizer update rule and how the multi-timescale loss in Eq. (4) is used during training.
[IV (Corridor dataset)] The dataset description gives the total number of frames but not the distribution of abnormal frames across activity classes; a per-class frame count would help readers assess the difficulty and potential biases.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the multi-timescale anomaly score is an empirically evaluated prediction-error pipeline, not a reduction of the result into its inputs.

full rationale

The claimed derivation chain is: supervised intermediate convolutional layers produce pose predictions at timescales {3, 5, 13, 25}; prediction errors are weighted by pose confidence and combined as layer losses; the test-time anomaly score in Eq. (7) is the uniform average of per-layer errors; and Frame-AUC is evaluated against external frame-level annotations on HR-ShanghaiTech, HR-Avenue, and the new Corridor dataset. None of these steps defines the target outcome in terms of itself: the model is trained only on normal data (Section V-B), the anomaly labels are external annotations, and the reported scores are computed on held-out test sets. The choice S = {3, 5, 13, 25} and the uniform average in Eq. (7) are hand-picked hyperparameters rather than quantities fitted to the benchmark labels; selecting the configuration after inspecting Table III is a test-set reuse or robustness concern, not a circularity. The acknowledged degradation on HR-Avenue with the 25-step timescale is an empirical caveat, not a circular reduction. No load-bearing self-citation is present: the HR dataset variants and pose preprocessing are attributed to prior work by Morais et al. and OpenPose, which are independent external sources. I therefore find no circular step.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method is a learned predictor, so its central claim rests on the availability of representative normal training data, accurate pose trajectories, and the validity of prediction error as an anomaly signal. The multi-timescale component rests on a hand-picked set of horizons and a uniform voting rule, both of which are parameters of the design rather than derived quantities.

free parameters (4)

Timescale set = {3, 5, 13, 25}
Hand-picked set of prediction horizons; no search or justification provided. The ablation in Table III shows that adding the 25-step timescale reduces HR-Avenue AUC, so the set is not uniformly optimal.
Anomaly detection threshold = not reported
A pre-defined threshold is used to tag abnormal frames (Section III-D), but its value and selection procedure are not given, making the reported AUC sensitive to this choice.
Model capacity hyperparameters = 7 layers, 1024 filters, filter lengths 3, 3, 5, 5, 5, 5, 5
Architecture size and filter configuration are chosen by hand with no ablation; the central result depends on this specific capacity.
Voting weights in Eq. (7) = uniform 1/|S|
Errors from different timescales are averaged equally; the paper does not learn or justify these weights, and its own ablation suggests equal weighting is not always optimal.

assumptions (4)

domain assumption The training set consists exclusively of normal activities and adequately represents the normal behavior seen at test time.
This is the standard setup for unsupervised anomaly detection stated in Section I and used in training (Section V-B). If normality is not representative, the prediction error baseline is invalid.
domain assumption Human pose trajectories extracted by the detector, tracker, and pose estimator are sufficiently accurate to capture abnormal activities.
Section V-A relies entirely on these off-the-shelf tools; noisy or missed poses could affect prediction errors and anomaly scores.
domain assumption Prediction error of future and past poses is a valid proxy for abnormality, i.e., normal activities are predictable and abnormal activities are not.
This is the core premise of the anomaly scoring in Section III-D; if abnormal activities are also predictable from pose, the method cannot detect them.
ad hoc to paper A 1D convolutional layer's receptive field length corresponds to the temporal horizon of dynamics, so supervising layers with receptive fields 3, 5, 13, and 25 yields predictions at those timescales.
Section III-B states this mapping directly; it is a design choice specific to this architecture and not otherwise validated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multi-timescale Trajectory Prediction for Abnormal Human Activity Detection." pith.science (2026). https://pith.science/paper/4ZTNNQHK

@misc{pith2026190804321,
  author       = {Pith},
  title        = {Pith review of: Multi-timescale Trajectory Prediction for Abnormal Human Activity Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ZTNNQHK}},
  note         = {Machine review of arXiv:1908.04321}
}

read the original abstract

A classical approach to abnormal activity detection is to learn a representation for normal activities from the training data and then use this learned representation to detect abnormal activities while testing. Typically, the methods based on this approach operate at a fixed timescale - either a single time-instant (eg. frame-based) or a constant time duration (eg. video-clip based). But human abnormal activities can take place at different timescales. For example, jumping is a short term anomaly and loitering is a long term anomaly in a surveillance scenario. A single and pre-defined timescale is not enough to capture the wide range of anomalies occurring with different time duration. In this paper, we propose a multi-timescale model to capture the temporal dynamics at different timescales. In particular, the proposed model makes future and past predictions at different timescales for a given input pose trajectory. The model is multi-layered where intermediate layers are responsible to generate predictions corresponding to different timescales. These predictions are combined to detect abnormal activities. In addition, we also introduce an abnormal activity data-set for research use that contains 4,83,566 annotated frames. Data-set will be made available at https://rodrigues-royston.github.io/Multi-timescale_Trajectory_Prediction/ Our experiments show that the proposed model can capture the anomalies of different time duration and outperforms existing methods.

Figures

Figures reproduced from arXiv: 1908.04321 by the authors.

**Figure 1.** An illustration of how our model captures a long term anomaly. The anomaly in consideration is loitering - the intermediate frames are shown in the right figure with the person involved in red box. The plots in the left shows the prediction errors at different timescales. The prediction errors are less at lower timescales (3 and 5) because at these timescales, the model considers it as a normal activity (walking). T… view at source ↗

**Figure 2.** Top-level block diagram of the proposed framework. The future prediction model takes the input sequence and generate predictions at different timescales. To generate predictions at timescale 1, the model first splits the sequence into smaller sub-sequences and then makes future predictions for these sub-sequences. These predictions are combined to get the future prediction for the input sequence at this timescale. T… view at source ↗

**Figure 3.** It demonstrates higher prediction errors for abnormal [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 3.** Figure 3: An illustration of predicted poses from the proposed model. The first scene has a walking action while the other three have abnormal actions. The green and blue poses represent the actual and predicted poses, respectively. The model generate large prediction errors for…

**Figure 4.** Figure 4: The detailed architecture of the proposed model for future prediction. X1-XT is the input pose trajectory. After encoding, the vectors f1-fT are passed to a series of 1D convolutional filters. A few intermediate layers generate predictions at different timescales. To g…

**Figure 5.** Figure 5: Computation of node and layer losses at a particular timescale. To compute the prediction loss generated by the node n1, we simply add the errors e(1, 1) to e(1, 5) because n1 makes predictions for t1 to t5. We add all the node losses to get the total node loss. To com…

**Figure 6.** Figure 6: Samples images from the proposed Corridor dataset. Dataset Training frames Testing frames Abnormal activities ShanghaiTech [12] 2,74,515 42,883 Throwing Object, Jumping, Pushing Riding a Bike, Loitering, Climbing USC Ped-1 [8] 6,800 7,200 Bikers, small carts, walking a…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 16 canonical work pages

[1]

Abati, A

D. Abati, A. Porrello, S. Calderara, and R. Cucchiara. Latent space autoregression for novelty detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019

work page 2019
[2]

A. Adam, E. Rivlin, I. Shimshoni, and D. Reinitz. Robust real-time unusual event detection using multiple ﬁxed-location monitors. IEEE transactions on pattern analysis and machine intelligence , 30(3):555– 560, 2008

work page 2008
[3]

Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y . Sheikh. OpenPose: realtime multi-person 2D pose estimation using Part Afﬁnity Fields. In arXiv preprint arXiv:1812.08008 , 2018

arXiv 2018
[4]

Z. Cao, T. Simon, S.-E. Wei, and Y . Sheikh. Realtime multi-person 2d pose estimation using part afﬁnity ﬁelds. In CVPR, 2017

work page 2017
[5]

Hasan, J

M. Hasan, J. Choi, J. Neumann, A. K. Roy-Chowdhury, and L. S. Davis. Learning temporal regularity in video sequences. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 733–742, 2016

work page 2016
[6]

Hinami, T

R. Hinami, T. Mei, and S. Satoh. Joint detection and recounting of abnormal events by learning deep generic knowledge. In Proceedings of the IEEE International Conference on Computer Vision , pages 3619– 3627, 2017

work page 2017
[7]

Huang, V

J. Huang, V . Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y . Song, S. Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7310– 7311, 2017

work page 2017
[8]

W. Li, V . Mahadevan, and N. Vasconcelos. Anomaly detection and localization in crowded scenes. IEEE transactions on pattern analysis and machine intelligence , 36(1):18–32, 2013

work page 2013

Show all 18 references

[9]

W. Liu, W. Luo, D. Lian, and S. Gao. Future frame prediction for anomaly detection–a new baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6536– 6545, 2018

2018
[10]

C. Long, A. Haizhou, Z. Zijie, and S. Chong. Real-time multiple people tracking with deeply learned candidate selection and person re- identiﬁcation. In ICME, 2018

2018
[11]

C. Lu, J. Shi, and J. Jia. Abnormal event detection at 150 fps in matlab. In Proceedings of the IEEE international conference on computer vision, pages 2720–2727, 2013

2013
[12]

W. Luo, W. Liu, and S. Gao. A revisit of sparse coding based anomaly detection in stacked rnn framework. In Proceedings of the IEEE International Conference on Computer Vision , pages 341–349, 2017

2017
[13]

Morais, V

R. Morais, V . Le, T. Tran, B. Saha, M. Mansour, and S. Venkatesh. Learning regularity in skeleton trajectories for anomaly detection in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11996–12004, 2019

2019
[14]

Najibi, P

M. Najibi, P. Samangouei, R. Chellappa, and L. Davis. SSH: Single stage headless face detector. In The IEEE International Conference on Computer Vision (ICCV) , 2017

2017
[15]

Ravanbakhsh, M

M. Ravanbakhsh, M. Nabi, E. Sangineto, L. Marcenaro, C. Regazzoni, and N. Sebe. Abnormal event detection in videos using generative adversarial nets. In 2017 IEEE International Conference on Image Processing (ICIP), pages 1577–1581. IEEE, 2017

2017
[16]

H. T. Tran and D. Hogg. Anomaly detection using a convolutional winner-take-all autoencoder. In Proceedings of the British Machine Vision Conference 2017. British Machine Vision Association, 2017

2017
[17]

Tudor Ionescu, S

R. Tudor Ionescu, S. Smeureanu, B. Alexe, and M. Popescu. Unmasking the abnormal events in video. In Proceedings of the IEEE International Conference on Computer Vision , pages 2895–2903, 2017

2017
[18]

D. Xu, E. Ricci, Y . Yan, J. Song, and N. Sebe. Learning deep representations of appearance and motion for anomalous event detection. arXiv preprint arXiv:1510.01553 , 2015

2015 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Abati, A

D. Abati, A. Porrello, S. Calderara, and R. Cucchiara. Latent space autoregression for novelty detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019

work page 2019

[2] [2]

A. Adam, E. Rivlin, I. Shimshoni, and D. Reinitz. Robust real-time unusual event detection using multiple ﬁxed-location monitors. IEEE transactions on pattern analysis and machine intelligence , 30(3):555– 560, 2008

work page 2008

[3] [3]

Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y . Sheikh. OpenPose: realtime multi-person 2D pose estimation using Part Afﬁnity Fields. In arXiv preprint arXiv:1812.08008 , 2018

arXiv 2018

[4] [4]

Z. Cao, T. Simon, S.-E. Wei, and Y . Sheikh. Realtime multi-person 2d pose estimation using part afﬁnity ﬁelds. In CVPR, 2017

work page 2017

[5] [5]

Hasan, J

M. Hasan, J. Choi, J. Neumann, A. K. Roy-Chowdhury, and L. S. Davis. Learning temporal regularity in video sequences. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 733–742, 2016

work page 2016

[6] [6]

Hinami, T

R. Hinami, T. Mei, and S. Satoh. Joint detection and recounting of abnormal events by learning deep generic knowledge. In Proceedings of the IEEE International Conference on Computer Vision , pages 3619– 3627, 2017

work page 2017

[7] [7]

Huang, V

J. Huang, V . Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y . Song, S. Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7310– 7311, 2017

work page 2017

[8] [8]

W. Li, V . Mahadevan, and N. Vasconcelos. Anomaly detection and localization in crowded scenes. IEEE transactions on pattern analysis and machine intelligence , 36(1):18–32, 2013

work page 2013