REVIEW 2 major objections 5 minor 2 cited by
Learning Trajectory Dependencies for Human Motion Prediction
T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A feed-forward network that predicts residual DCT coefficients in trajectory space, with learned graph connectivity, outperforms recurrent and convolutional baselines for human motion prediction.
desk verdict Solid, reproducible motion-prediction paper with a genuinely useful DCT-in-trajectory trick and good ablations, but the 'SOTA in all cases' claim needs tempering because the strongest adversarial baseline is absent from 3D tables. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the trajectory-space DCT representation: each joint's position or angle over time is expanded in Discrete Cosine Transform bases, and the network operates on the resulting coefficients rather than on raw frames. The graph-convolution layer with a learnable weighted adjacency matrix $A^{(p)}$ per layer is the mechanism that lets the network choose which joints exchange information; no pre-defined skeleton graph is used. Residual prediction, obtained by padding the observed sequence with copies of the last frame and predicting only the difference in coefficient space, carries the zero-velocity baseline into frequency space.
What would settle it
Run the same network on a motion sequence with sharp, high-frequency joint excursions, such as a fast kick or a stumble, truncate the DCT coefficients as prescribed, and compare MPJPE against the same model with all coefficients retained; if truncation consistently degrades accuracy on such non-smooth motions, the smoothness premise fails.
Extended reading notes
Core claim
The paper's central claim is that motion prediction can be reformulated as residual prediction in trajectory space: a feed-forward network maps the DCT coefficients of the observed, last-pose-padded trajectory to the coefficients of the true future sequence, and inverse DCT turns the output back into poses. Because human motion in 3D coordinates is smooth, truncating high-frequency DCT coefficients acts as a built-in smoothing prior, preventing the jitter that comes from frame-by-frame regression. The same network treats the body as a fully connected graph with a trainable adjacency matrix per layer, allowing it to discover joint dependencies beyond the kinematic tree. With this design, the paper reports the lowest mean-per-joint position error on all three benchmarks for both short-term and long-term horizons, especially when the model is trained directly on 3D joint positions.
Load-bearing premise
The result rests on the premise that the temporal trajectory of each joint is smooth enough that a small set of low-frequency DCT coefficients retains the predictive signal, which the paper's own supplement shows holds for 3D coordinates but fails for angle trajectories.
Editorial extensions
If this is right
- Recurrent models can be replaced by a feed-forward network with a fixed observation window, removing the need to manage recurrent state and the error accumulation that comes with it.
- Training on 3D coordinates gives lower position error than training on joint angles, and angle-space errors can mislead evaluation because different angle sets can yield the same 3D pose; future protocols should report 3D error directly.
- Truncating high-frequency DCT coefficients acts as a smoothing regularizer, so the same architecture can trade off detail and jitter by choosing how many coefficients to keep.
- A per-layer learned adjacency matrix captures cross-limb dependencies that a fixed skeletal graph would miss, improving accuracy on actions that require synchronized body parts.
Reading between the lines
- The same trajectory-space recipe should transfer to other smooth articulated-motion forecasting tasks, such as hand tracking or animal locomotion, wherever per-joint trajectories are reasonably smooth.
- The learned adjacency matrices can be inspected after training to reveal whether the network rediscovers known anatomical couplings or discovers action-specific joint dependencies.
- A natural extension is to make the number of retained DCT coefficients adaptive per joint or per action, since the paper's own supplement shows angle trajectories require many more coefficients than 3D trajectories.
- The reported angle-space failures suggest a testable hypothesis for future benchmarks: comparing methods in angle space systematically penalizes models whose predictions are geometrically correct but parameterized differently.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a feed-forward network for human motion prediction that encodes the observed pose sequence in trajectory space via the Discrete Cosine Transform (DCT) and models spatial joint dependencies with graph convolutional layers whose adjacency matrices are learned during training. A residual formulation is used: the network predicts DCT-coefficient residuals relative to the DCT coefficients of the observed sequence padded by replicating the last pose, motivated by the zero-velocity baseline of [17]. The method is evaluated on Human3.6M, CMU-Mocap, and 3DPW, in both joint-angle and 3D coordinate representations, and is reported to outperform prior RNN and convolutional baselines. The paper also argues that angle-space evaluation is unreliable because multiple angle sets can map to the same 3D pose, and that training directly on 3D coordinates gives the best results.
Significance. If the results hold, the paper makes a useful contribution to human motion prediction: it provides a simple feed-forward alternative to RNN-based methods, shows that DCT-based trajectory encoding removes the need to manually set temporal convolutional filter sizes, and demonstrates that learning graph connectivity is preferable to using a fixed kinematic-tree graph. The ablations in Tables 6 and 7 individually justify the DCT encoding, the replicate-padding strategy, residual connections, and the learned connectivity. The supplementary sensitivity analysis of the number of DCT coefficients is informative and honestly reports that angle trajectories are not smooth and therefore require full-frequency DCT, which confines the benefit of coefficient truncation to smooth 3D trajectories; the stress-test concern about non-smooth trajectories is thus already addressed by the manuscript's own analysis.
major comments (2)
- [Abstract and Section 1] The abstract states that the proposed approach 'achieves state of the art performance' and Section 1 claims 'state-of-the-art results in all cases.' These blanket claims are contradicted by Table 3: in long-term joint-angle prediction at 1000 ms, the average error of the proposed method is 1.27, which is higher than AGED w/adv's 1.09. While the authors acknowledge this in Section 4.3, the abstract and introduction remain unqualified. Please revise the claims to specify that the state-of-the-art result holds for short-term angle prediction and for 3D-coordinate prediction, with the caveat that the 3D comparison set is limited as described in the next comment.
- [Section 4.2 and Tables 2–5] The 3D-coordinate comparisons omit AGED [9], the strongest prior method in angle space, because the authors state that its code and angle predictions are unavailable. Consequently, the statement in Section 4.3 that 'we outperform the baselines by a large margin, particularly when training directly on 3D' is supported only against Residual sup. [17] and convSeq2Seq [16], both adapted by the authors to accept 3D input. This does not establish state-of-the-art 3D prediction against the full field. The authors should either soften the claim to 'outperform the compared baselines' or make a stronger effort to include AGED in 3D (for example, by obtaining or converting AGED's published angle outputs if they become available), or clearly state the comparison scope in the abstract and conclusion.
minor comments (5)
- [Section 3.4 and Section 4.2] Section 3.4 says training is performed in the exponential map representation of joint angles, while Section 4.2 says the evaluation metric is Euclidean distance in Euler angle representation. Please clarify whether the exponential-map predictions are converted to Euler angles before computing the error, or whether this is a typo.
- [Tables 1–5] No uncertainty estimates (standard deviations, confidence intervals, or repeated runs) are reported. Please state whether the reported numbers are from a single training run or averaged, and consider adding error bars to the key comparisons.
- [Table 7] The 'pre-defined connectivity' baseline used in Table 7 is not fully specified; it is unclear whether it uses the kinematic tree with a normalized adjacency matrix, a full graph, or some other construction. Please provide the exact definition so the ablation is reproducible.
- [Section 4.1] The sentence 'to predict the future10 frames' is missing a space; it should read 'future 10 frames.'
- [Table 6] The checkmark notation in Table 6 is not immediately self-explanatory; a legend or a sentence stating which components are active in each row would improve readability.
Circularity Check
No significant circularity: the DCT targets are ground-truth-derived, the residual formulation is externally motivated, and the learned components are trained on standard splits.
full rationale
I walked the paper's derivation chain from the DCT temporal encoding (Eq. 1), through the residual formulation in Section 3.1, the graph-convolutional layer (Eq. 4), and the losses used for training (Eqs. 5 and 6). The target DCT coefficients are computed from the ground-truth future sequence X_{1:N+T}, not from the model's own output, and the input is the observed sequence padded with the last pose; the residual is taken against this fixed zero-velocity-style baseline, which the paper explicitly motivates by the external result in [17]. No fitted parameter is subsequently renamed as a prediction: the learnable adjacency matrices A^(p) are trained on the training split, and the number of DCT coefficients is selected using a validation split (supplementary Section 4.2). The paper contains no load-bearing self-citations; the cited trajectory-space idea [1], graph-convolution machinery [13, 22], and baselines [16, 17] are all external prior work, and no uniqueness theorem is invoked to force the architectural choice. The limitations noted by the reader, such as the absence of AGED in the 3D tables, the lack of variance estimates, and the angle-space 1000 ms result in Table 3, concern the completeness and statistical robustness of the comparison rather than any reduction of a prediction to an input. I therefore cannot exhibit any specific circular step, and the derivation is self-contained with respect to the concerns enumerated in the review rules.
Assumptions & free parameters
free parameters (8)
- Number of input DCT coefficients, short-term 3D =
15
- Number of input DCT coefficients, long-term 3D =
30
- Number of input DCT coefficients, angle representation =
20 short-term, 35 long-term
- Graph feature width =
256
- Number of residual graph blocks =
12
- Learning rate and schedule =
0.0005 with 0.96 decay every 2 epochs
- Batch size =
16
- Training epochs =
50
assumptions (4)
- domain assumption Joint trajectories are smooth enough for a truncated DCT to preserve the predictive information.
- domain assumption A fully connected graph with a per-layer learnable adjacency matrix can represent the spatial dependencies needed for motion prediction.
- domain assumption Padding the observed sequence with copies of the last pose is a valid way to make the input and target sequences the same length for DCT residual learning.
- standard math The DCT and IDCT under the normalization of Eqs. 1 and 3 are invertible when all coefficients are kept.
Cite this review
Pith. "Pith review of Learning Trajectory Dependencies for Human Motion Prediction." pith.science (2026). https://pith.science/paper/RPKY2RSN
@misc{pith2026190805436,
author = {Pith},
title = {Pith review of: Learning Trajectory Dependencies for Human Motion Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/RPKY2RSN}},
note = {Machine review of arXiv:1908.05436}
}
read the original abstract
Human motion prediction, i.e., forecasting future body poses given observed pose sequence, has typically been tackled with recurrent neural networks (RNNs). However, as evidenced by prior work, the resulted RNN models suffer from prediction errors accumulation, leading to undesired discontinuities in motion prediction. In this paper, we propose a simple feed-forward deep network for motion prediction, which takes into account both temporal smoothness and spatial dependencies among human body joints. In this context, we then propose to encode temporal information by working in trajectory space, instead of the traditionally-used pose space. This alleviates us from manually defining the range of temporal dependencies (or temporal convolutional filter size, as done in previous work). Moreover, spatial dependency of human pose is encoded by treating a human pose as a generic graph (rather than a human skeletal kinematic tree) formed by links between every pair of body joints. Instead of using a pre-defined graph structure, we design a new graph convolutional network to learn graph connectivity automatically. This allows the network to capture long range dependencies beyond that of human kinematic tree. We evaluate our approach on several standard benchmark datasets for motion prediction, including Human3.6M, the CMU motion capture dataset and 3DPW. Our experiments clearly demonstrate that the proposed approach achieves state of the art performance, and is applicable to both angle-based and position-based pose representations. The code is available at https://github.com/wei-mao-2019/LearnTrajDep
Figures
Figures from the paper (9 more)
Forward citations
Cited by 2 Pith papers
-
Agent-Centric Animal Pose Forecasting
Agent-centric transformers trained through a composable library reproduce several marginal statistics of courting fly behavior, but discriminators still separate simulated from real flies.
-
Autonomous Telerehabilitation via Skeletal Motion Prediction and Joint-Level Performance Assessment
A two-module RGB-video rehabilitation pipeline is proposed, with a 96.45% squat-quality classifier and an adopted STARS predictor, but no end-to-end or clinical validation is performed.
Reference graph
Works this paper leans on
-
[17]
Julieta Martinez, Michael J. Black, and Javier Romero. On human motion prediction using recurrent neural networks. In CVPR, July 2017. 1, 2, 4, 5, 6, 7, 9, 10, 11, 12
work page 2017
-
[9]
Adversarial geometry-aware human mo- tion prediction
Liang-Yan Gui, Yu-Xiong Wang, Xiaodan Liang, and Jos´e MF Moura. Adversarial geometry-aware human mo- tion prediction. In ECCV, pages 786–803, 2018. 1, 2, 5, 6, 7, 9
work page 2018
-
[16]
Con- volutional sequence to sequence model for human dynamics
Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee. Con- volutional sequence to sequence model for human dynamics. In CVPR, pages 5226–5234, 2018. 1, 2, 5, 6, 7, 9, 10, 11
work page 2018
-
[1]
Nonrigid structure from motion in trajectory space
Ijaz Akhter, Yaser Sheikh, Sohaib Khan, and Takeo Kanade. Nonrigid structure from motion in trajectory space. In Ad- vances in neural information processing systems, pages 41– 48, 2009. 2, 3
work page 2009
-
[2]
Towards principled meth- ods for training generative adversarial networks
Mart ´ın Arjovsky and L´eon Bottou. Towards principled meth- ods for training generative adversarial networks. In ICLR,
-
[3]
Matthew Brand and Aaron Hertzmann. Style machines. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques , pages 183–192. ACM Press/Addison-Wesley Publishing Co., 2000. 1
work page 2000
-
[4]
Spectral networks and locally connected networks on graphs
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Le- Cun. Spectral networks and locally connected networks on graphs. In ICLR, 2014. 2
work page 2014
-
[5]
Black, Danica Kragic, and Hed- vig Kjellstrom
Judith Butepage, Michael J. Black, Danica Kragic, and Hed- vig Kjellstrom. Deep representation learning for human mo- tion prediction and classification. In CVPR, July 2017. 1, 2, 3
work page 2017
Show all 25 references
-
[6]
Convolutional neural networks on graphs with fast localized spectral filtering
Micha ¨el Defferrard, Xavier Bresson, and Pierre Van- dergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural infor- mation processing systems, pages 3844–3852, 2016. 2
2016
-
[7]
Recurrent network models for human dynam- ics
Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Ji- tendra Malik. Recurrent network models for human dynam- ics. In ICCV, pages 4346–4354, 2015. 1, 2
2015
-
[8]
Multi-hypothesis motion planning for visual object tracking
Haifeng Gong, Jack Sim, Maxim Likhachev, and Jianbo Shi. Multi-hypothesis motion planning for visual object tracking. In ICCV, pages 619–626. IEEE, 2011. 1
2011
-
[10]
Human3.6m: Large scale datasets and predic- tive methods for 3d human sensing in natural environments
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predic- tive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 36(7):1325–1339, jul 2014. 4, 5, 6
2014
-
[11]
Structural-rnn: Deep learning on spatio- temporal graphs
Ashesh Jain, Amir Roshan Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio- temporal graphs. In CVPR, pages 5308–5317, 2016. 1, 2
2016
-
[12]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,
-
[13]
Semi-supervised classifi- cation with graph convolutional networks
Thomas N Kipf and Max Welling. Semi-supervised classifi- cation with graph convolutional networks. In ICLR, 2017. 2, 4
2017
-
[14]
Skip-thought vectors
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302, 2015. 2
2015
-
[15]
Anticipating human activities for reactive robotic response
Hema Swetha Koppula and Ashutosh Saxena. Anticipating human activities for reactive robotic response. InIROS, page
-
[18]
A survey of motion planning and con- trol techniques for self-driving urban vehicles
Brian Paden, Michal ˇC´ap, Sze Zheng Yong, Dmitry Yershov, and Emilio Frazzoli. A survey of motion planning and con- trol techniques for self-driving urban vehicles. IEEE Trans- actions on intelligent vehicles, 1(1):33–55, 2016. 1
2016
-
[19]
On the difficulty of training recurrent neural networks
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. InICML, pages 1310–1318, 2013. 1
2013
-
[20]
Automatic differentiation in pytorch
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- ban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017. 6, 11
2017
-
[21]
Gen- erating text with recurrent neural networks
Ilya Sutskever, James Martens, and Geoffrey E Hinton. Gen- erating text with recurrent neural networks. In ICML, pages 1017–1024, 2011. 2
2011
-
[22]
Graph at- tention networks
Petar Veli ˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph at- tention networks. In ICLR, 2018. 2, 3, 4
2018
-
[23]
Recovering accurate 3d human pose in the wild using imus and a moving camera
Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In ECCV, 2018. 5
2018
-
[24]
Gaus- sian process dynamical models for human motion
Jack M Wang, David J Fleet, and Aaron Hertzmann. Gaus- sian process dynamical models for human motion. IEEE transactions on pattern analysis and machine intelligence , 30(2):283–298, 2008. 1
2008
-
[25]
Spatial tempo- ral graph convolutional networks for skeleton-based action recognition
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial tempo- ral graph convolutional networks for skeleton-based action recognition. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. 3
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.