Pith. sign in

REVIEW 2 major objections 5 minor 2 cited by

Learning Trajectory Dependencies for Human Motion Prediction

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A feed-forward network that predicts residual DCT coefficients in trajectory space, with learned graph connectivity, outperforms recurrent and convolutional baselines for human motion prediction.

desk verdict Solid, reproducible motion-prediction paper with a genuinely useful DCT-in-trajectory trick and good ablations, but the 'SOTA in all cases' claim needs tempering because the strongest adversarial baseline is absent from 3D tables. read the letter →

arxiv 1908.05436 v3 pith:RPKY2RSN submitted 2019-08-15 cs.CV

classification cs.CV
keywords humanmotionpredictiondiscretecosinetransformtrajectoryspacegraphconvolutionalnetworklearnableconnectivity3Dposerepresentationresidualforecasting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Human motion prediction has usually been built on recurrent neural networks, which can accumulate errors and produce jerky transitions. This paper proposes a compact feed-forward alternative: encode each joint's trajectory with Discrete Cosine Transform coefficients, predict residual coefficients for the future, and decode them back to poses. Spatial structure is handled by graph-convolution layers that learn their own connectivity, instead of assuming the skeleton's kinematic tree. The authors report state-of-the-art accuracy on three standard benchmarks, with the best results when training directly on 3D coordinates, and argue that angle-space evaluation is unreliable because different angle sets can describe the same pose.

What carries the argument

The load-bearing object is the trajectory-space DCT representation: each joint's position or angle over time is expanded in Discrete Cosine Transform bases, and the network operates on the resulting coefficients rather than on raw frames. The graph-convolution layer with a learnable weighted adjacency matrix $A^{(p)}$ per layer is the mechanism that lets the network choose which joints exchange information; no pre-defined skeleton graph is used. Residual prediction, obtained by padding the observed sequence with copies of the last frame and predicting only the difference in coefficient space, carries the zero-velocity baseline into frequency space.

What would settle it

Run the same network on a motion sequence with sharp, high-frequency joint excursions, such as a fast kick or a stumble, truncate the DCT coefficients as prescribed, and compare MPJPE against the same model with all coefficients retained; if truncation consistently degrades accuracy on such non-smooth motions, the smoothness premise fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that motion prediction can be reformulated as residual prediction in trajectory space: a feed-forward network maps the DCT coefficients of the observed, last-pose-padded trajectory to the coefficients of the true future sequence, and inverse DCT turns the output back into poses. Because human motion in 3D coordinates is smooth, truncating high-frequency DCT coefficients acts as a built-in smoothing prior, preventing the jitter that comes from frame-by-frame regression. The same network treats the body as a fully connected graph with a trainable adjacency matrix per layer, allowing it to discover joint dependencies beyond the kinematic tree. With this design, the paper reports the lowest mean-per-joint position error on all three benchmarks for both short-term and long-term horizons, especially when the model is trained directly on 3D joint positions.

Load-bearing premise

The result rests on the premise that the temporal trajectory of each joint is smooth enough that a small set of low-frequency DCT coefficients retains the predictive signal, which the paper's own supplement shows holds for 3D coordinates but fails for angle trajectories.

Editorial extensions

If this is right

  • Recurrent models can be replaced by a feed-forward network with a fixed observation window, removing the need to manage recurrent state and the error accumulation that comes with it.
  • Training on 3D coordinates gives lower position error than training on joint angles, and angle-space errors can mislead evaluation because different angle sets can yield the same 3D pose; future protocols should report 3D error directly.
  • Truncating high-frequency DCT coefficients acts as a smoothing regularizer, so the same architecture can trade off detail and jitter by choosing how many coefficients to keep.
  • A per-layer learned adjacency matrix captures cross-limb dependencies that a fixed skeletal graph would miss, improving accuracy on actions that require synchronized body parts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same trajectory-space recipe should transfer to other smooth articulated-motion forecasting tasks, such as hand tracking or animal locomotion, wherever per-joint trajectories are reasonably smooth.
  • The learned adjacency matrices can be inspected after training to reveal whether the network rediscovers known anatomical couplings or discovers action-specific joint dependencies.
  • A natural extension is to make the number of retained DCT coefficients adaptive per joint or per action, since the paper's own supplement shows angle trajectories require many more coefficients than 3D trajectories.
  • The reported angle-space failures suggest a testable hypothesis for future benchmarks: comparing methods in angle space systematically penalizes models whose predictions are geometrically correct but parameterized differently.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a feed-forward network for human motion prediction that encodes the observed pose sequence in trajectory space via the Discrete Cosine Transform (DCT) and models spatial joint dependencies with graph convolutional layers whose adjacency matrices are learned during training. A residual formulation is used: the network predicts DCT-coefficient residuals relative to the DCT coefficients of the observed sequence padded by replicating the last pose, motivated by the zero-velocity baseline of [17]. The method is evaluated on Human3.6M, CMU-Mocap, and 3DPW, in both joint-angle and 3D coordinate representations, and is reported to outperform prior RNN and convolutional baselines. The paper also argues that angle-space evaluation is unreliable because multiple angle sets can map to the same 3D pose, and that training directly on 3D coordinates gives the best results.

Significance. If the results hold, the paper makes a useful contribution to human motion prediction: it provides a simple feed-forward alternative to RNN-based methods, shows that DCT-based trajectory encoding removes the need to manually set temporal convolutional filter sizes, and demonstrates that learning graph connectivity is preferable to using a fixed kinematic-tree graph. The ablations in Tables 6 and 7 individually justify the DCT encoding, the replicate-padding strategy, residual connections, and the learned connectivity. The supplementary sensitivity analysis of the number of DCT coefficients is informative and honestly reports that angle trajectories are not smooth and therefore require full-frequency DCT, which confines the benefit of coefficient truncation to smooth 3D trajectories; the stress-test concern about non-smooth trajectories is thus already addressed by the manuscript's own analysis.

major comments (2)
  1. [Abstract and Section 1] The abstract states that the proposed approach 'achieves state of the art performance' and Section 1 claims 'state-of-the-art results in all cases.' These blanket claims are contradicted by Table 3: in long-term joint-angle prediction at 1000 ms, the average error of the proposed method is 1.27, which is higher than AGED w/adv's 1.09. While the authors acknowledge this in Section 4.3, the abstract and introduction remain unqualified. Please revise the claims to specify that the state-of-the-art result holds for short-term angle prediction and for 3D-coordinate prediction, with the caveat that the 3D comparison set is limited as described in the next comment.
  2. [Section 4.2 and Tables 2–5] The 3D-coordinate comparisons omit AGED [9], the strongest prior method in angle space, because the authors state that its code and angle predictions are unavailable. Consequently, the statement in Section 4.3 that 'we outperform the baselines by a large margin, particularly when training directly on 3D' is supported only against Residual sup. [17] and convSeq2Seq [16], both adapted by the authors to accept 3D input. This does not establish state-of-the-art 3D prediction against the full field. The authors should either soften the claim to 'outperform the compared baselines' or make a stronger effort to include AGED in 3D (for example, by obtaining or converting AGED's published angle outputs if they become available), or clearly state the comparison scope in the abstract and conclusion.
minor comments (5)
  1. [Section 3.4 and Section 4.2] Section 3.4 says training is performed in the exponential map representation of joint angles, while Section 4.2 says the evaluation metric is Euclidean distance in Euler angle representation. Please clarify whether the exponential-map predictions are converted to Euler angles before computing the error, or whether this is a typo.
  2. [Tables 1–5] No uncertainty estimates (standard deviations, confidence intervals, or repeated runs) are reported. Please state whether the reported numbers are from a single training run or averaged, and consider adding error bars to the key comparisons.
  3. [Table 7] The 'pre-defined connectivity' baseline used in Table 7 is not fully specified; it is unclear whether it uses the kinematic tree with a normalized adjacency matrix, a full graph, or some other construction. Please provide the exact definition so the ablation is reproducible.
  4. [Section 4.1] The sentence 'to predict the future10 frames' is missing a space; it should read 'future 10 frames.'
  5. [Table 6] The checkmark notation in Table 6 is not immediately self-explanatory; a legend or a sentence stating which components are active in each row would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the DCT targets are ground-truth-derived, the residual formulation is externally motivated, and the learned components are trained on standard splits.

full rationale

I walked the paper's derivation chain from the DCT temporal encoding (Eq. 1), through the residual formulation in Section 3.1, the graph-convolutional layer (Eq. 4), and the losses used for training (Eqs. 5 and 6). The target DCT coefficients are computed from the ground-truth future sequence X_{1:N+T}, not from the model's own output, and the input is the observed sequence padded with the last pose; the residual is taken against this fixed zero-velocity-style baseline, which the paper explicitly motivates by the external result in [17]. No fitted parameter is subsequently renamed as a prediction: the learnable adjacency matrices A^(p) are trained on the training split, and the number of DCT coefficients is selected using a validation split (supplementary Section 4.2). The paper contains no load-bearing self-citations; the cited trajectory-space idea [1], graph-convolution machinery [13, 22], and baselines [16, 17] are all external prior work, and no uniqueness theorem is invoked to force the architectural choice. The limitations noted by the reader, such as the absence of AGED in the 3D tables, the lack of variance estimates, and the angle-space 1000 ms result in Table 3, concern the completeness and statistical robustness of the comparison rather than any reduction of a prediction to an input. I therefore cannot exhibit any specific circular step, and the derivation is self-contained with respect to the concerns enumerated in the review rules.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The method is a supervised architecture, so its trainable weights are not enumerated as free parameters; the listed parameters are the hand-set and validation-chosen hyperparameters that the reported performance depends on. The claims also rest on empirical smoothness and graph-structure assumptions, and on standard mathematical properties of the DCT. No new physical entities are introduced.

free parameters (8)
  • Number of input DCT coefficients, short-term 3D = 15
    Chosen in Supplementary Section 4.2 on H3.6M validation to balance reconstruction error and smoothness.
  • Number of input DCT coefficients, long-term 3D = 30
    Same supplementary analysis; more coefficients are used for the longer 35-frame window.
  • Number of input DCT coefficients, angle representation = 20 short-term, 35 long-term
    Supplementary Section 4.2: angle trajectories are non-smooth, so the model keeps full or nearly full frequency content.
  • Graph feature width = 256
    Section 3.3: hidden dimension of every graph convolutional layer; hand-set architecture hyperparameter.
  • Number of residual graph blocks = 12
    Section 3.3: each block contains two graph convolutional layers; hand-set depth.
  • Learning rate and schedule = 0.0005 with 0.96 decay every 2 epochs
    Section 4.2 implementation details; standard chosen training hyperparameter.
  • Batch size = 16
    Section 4.2 implementation details.
  • Training epochs = 50
    Section 4.2 implementation details.
assumptions (4)
  • domain assumption Joint trajectories are smooth enough for a truncated DCT to preserve the predictive information.
    Supplementary Section 4 and Figs. 6-8: 15 DCT coefficients suffice for 3D coordinates, while angle trajectories show large jumps and need nearly all coefficients, so the assumption is representation-dependent.
  • domain assumption A fully connected graph with a per-layer learnable adjacency matrix can represent the spatial dependencies needed for motion prediction.
    Section 3.2; the network learns connectivity instead of using the kinematic tree, validated empirically in Table 7.
  • domain assumption Padding the observed sequence with copies of the last pose is a valid way to make the input and target sequences the same length for DCT residual learning.
    Section 3.1; motivated by the zero-velocity baseline of [17] and shown to help in the ablation of Table 6.
  • standard math The DCT and IDCT under the normalization of Eqs. 1 and 3 are invertible when all coefficients are kept.
    Standard orthogonality of DCT-II basis functions; the paper relies on this when reconstructing poses from predicted coefficients.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Trajectory Dependencies for Human Motion Prediction." pith.science (2026). https://pith.science/paper/RPKY2RSN

@misc{pith2026190805436,
  author       = {Pith},
  title        = {Pith review of: Learning Trajectory Dependencies for Human Motion Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RPKY2RSN}},
  note         = {Machine review of arXiv:1908.05436}
}
read the original abstract

Human motion prediction, i.e., forecasting future body poses given observed pose sequence, has typically been tackled with recurrent neural networks (RNNs). However, as evidenced by prior work, the resulted RNN models suffer from prediction errors accumulation, leading to undesired discontinuities in motion prediction. In this paper, we propose a simple feed-forward deep network for motion prediction, which takes into account both temporal smoothness and spatial dependencies among human body joints. In this context, we then propose to encode temporal information by working in trajectory space, instead of the traditionally-used pose space. This alleviates us from manually defining the range of temporal dependencies (or temporal convolutional filter size, as done in previous work). Moreover, spatial dependency of human pose is encoded by treating a human pose as a generic graph (rather than a human skeletal kinematic tree) formed by links between every pair of body joints. Instead of using a pre-defined graph structure, we design a new graph convolutional network to learn graph connectivity automatically. This allows the network to capture long range dependencies beyond that of human kinematic tree. We evaluate our approach on several standard benchmark datasets for motion prediction, including Human3.6M, the CMU motion capture dataset and 3DPW. Our experiments clearly demonstrate that the proposed approach achieves state of the art performance, and is applicable to both angle-based and position-based pose representations. The code is available at https://github.com/wei-mao-2019/LearnTrajDep

Figures

Figures reproduced from arXiv: 1908.05436 by the authors.

Figure 1
Figure 1. Human motion prediction. The left frames correspond to the observations. From top to bottom, we show the ground truth, and predictions obtained by the methods of [17] and [16], and by our approach on joint angles and 3d coordinates. Our predictions better match the ground truth. walking and golf swing, more complicated ones are typi￾cally tackled using deep networks [7, 11, 5, 17, 9, 16]. Because of the temporal nat… view at source ↗
Figure 2
Figure 2. Network architecture. We first apply the DCT to encode temporal pose information in trajectory space. The DCT coefficients are treated as features input to graph convolutional layers. We use 12 blocks of graph convolutional layers with residual connections and two additional graph convolutional layers, one at the beginning and one at the end, to encode the temporal information and decode the features to the residual… view at source ↗
Figure 3
Figure 3. Qualitative comparison of short-term (“Smoking” and “Walking”) and long-term (“Walking Dog”) predictions on H3.6M. From [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Drawbacks of the angle-based representation. (a) Joint [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 1
Figure 1. Figure 1: Motion prediction in 3D space on the “basketball action [PITH_FULL_IMAGE:figures/full_fig_p009_1.png]
Figure 2
Figure 2. Figure 2: Motion prediction in 3D space on the “basketball signal” [PITH_FULL_IMAGE:figures/full_fig_p010_2.png]
Figure 3
Figure 3. Figure 3: Motion prediction in 3D space on the “directing traffic” [PITH_FULL_IMAGE:figures/full_fig_p010_3.png]
Figure 4
Figure 4. Figure 4: Temporal trajectory of the x coordinate of one joint re￾constructed using different number of DCT coefficients. accuracy. To evidence this, in [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Fully-connected Network Structure [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 8
Figure 8. Figure 8: The temporal trajectory of one joint angle reconstructed [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 7
Figure 7. Figure 7: Average 3D prediction error over 4 actions (“walk [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 9
Figure 9. Figure 9: Prediction up to 4 seconds for the Phoning action of Hu [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agent-Centric Animal Pose Forecasting

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Agent-centric transformers trained through a composable library reproduce several marginal statistics of courting fly behavior, but discriminators still separate simulated from real flies.

  2. Autonomous Telerehabilitation via Skeletal Motion Prediction and Joint-Level Performance Assessment

    cs.CV 2026-08 conditional novelty 4.0 of 10

    A two-module RGB-video rehabilitation pipeline is proposed, with a 96.45% squat-quality classifier and an adopted STARS predictor, but no end-to-end or clinical validation is performed.

Reference graph

Works this paper leans on

25 extracted references · 24 canonical work pages · cited by 2 Pith papers

  1. [17]

    Black, and Javier Romero

    Julieta Martinez, Michael J. Black, and Javier Romero. On human motion prediction using recurrent neural networks. In CVPR, July 2017. 1, 2, 4, 5, 6, 7, 9, 10, 11, 12

  2. [9]

    Adversarial geometry-aware human mo- tion prediction

    Liang-Yan Gui, Yu-Xiong Wang, Xiaodan Liang, and Jos´e MF Moura. Adversarial geometry-aware human mo- tion prediction. In ECCV, pages 786–803, 2018. 1, 2, 5, 6, 7, 9

  3. [16]

    Con- volutional sequence to sequence model for human dynamics

    Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee. Con- volutional sequence to sequence model for human dynamics. In CVPR, pages 5226–5234, 2018. 1, 2, 5, 6, 7, 9, 10, 11

  4. [1]

    Nonrigid structure from motion in trajectory space

    Ijaz Akhter, Yaser Sheikh, Sohaib Khan, and Takeo Kanade. Nonrigid structure from motion in trajectory space. In Ad- vances in neural information processing systems, pages 41– 48, 2009. 2, 3

  5. [2]

    Towards principled meth- ods for training generative adversarial networks

    Mart ´ın Arjovsky and L´eon Bottou. Towards principled meth- ods for training generative adversarial networks. In ICLR,

  6. [3]

    Style machines

    Matthew Brand and Aaron Hertzmann. Style machines. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques , pages 183–192. ACM Press/Addison-Wesley Publishing Co., 2000. 1

  7. [4]

    Spectral networks and locally connected networks on graphs

    Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Le- Cun. Spectral networks and locally connected networks on graphs. In ICLR, 2014. 2

  8. [5]

    Black, Danica Kragic, and Hed- vig Kjellstrom

    Judith Butepage, Michael J. Black, Danica Kragic, and Hed- vig Kjellstrom. Deep representation learning for human mo- tion prediction and classification. In CVPR, July 2017. 1, 2, 3

Show all 25 references
  1. [6]

    Convolutional neural networks on graphs with fast localized spectral filtering

    Micha ¨el Defferrard, Xavier Bresson, and Pierre Van- dergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural infor- mation processing systems, pages 3844–3852, 2016. 2

  2. [7]

    Recurrent network models for human dynam- ics

    Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Ji- tendra Malik. Recurrent network models for human dynam- ics. In ICCV, pages 4346–4354, 2015. 1, 2

  3. [8]

    Multi-hypothesis motion planning for visual object tracking

    Haifeng Gong, Jack Sim, Maxim Likhachev, and Jianbo Shi. Multi-hypothesis motion planning for visual object tracking. In ICCV, pages 619–626. IEEE, 2011. 1

  4. [10]

    Human3.6m: Large scale datasets and predic- tive methods for 3d human sensing in natural environments

    Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predic- tive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 36(7):1325–1339, jul 2014. 4, 5, 6

  5. [11]

    Structural-rnn: Deep learning on spatio- temporal graphs

    Ashesh Jain, Amir Roshan Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio- temporal graphs. In CVPR, pages 5308–5317, 2016. 1, 2

  6. [12]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

  7. [13]

    Semi-supervised classifi- cation with graph convolutional networks

    Thomas N Kipf and Max Welling. Semi-supervised classifi- cation with graph convolutional networks. In ICLR, 2017. 2, 4

  8. [14]

    Skip-thought vectors

    Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302, 2015. 2

  9. [15]

    Anticipating human activities for reactive robotic response

    Hema Swetha Koppula and Ashutosh Saxena. Anticipating human activities for reactive robotic response. InIROS, page

  10. [18]

    A survey of motion planning and con- trol techniques for self-driving urban vehicles

    Brian Paden, Michal ˇC´ap, Sze Zheng Yong, Dmitry Yershov, and Emilio Frazzoli. A survey of motion planning and con- trol techniques for self-driving urban vehicles. IEEE Trans- actions on intelligent vehicles, 1(1):33–55, 2016. 1

  11. [19]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. InICML, pages 1310–1318, 2013. 1

  12. [20]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- ban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017. 6, 11

  13. [21]

    Gen- erating text with recurrent neural networks

    Ilya Sutskever, James Martens, and Geoffrey E Hinton. Gen- erating text with recurrent neural networks. In ICML, pages 1017–1024, 2011. 2

  14. [22]

    Graph at- tention networks

    Petar Veli ˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph at- tention networks. In ICLR, 2018. 2, 3, 4

  15. [23]

    Recovering accurate 3d human pose in the wild using imus and a moving camera

    Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In ECCV, 2018. 5

  16. [24]

    Gaus- sian process dynamical models for human motion

    Jack M Wang, David J Fleet, and Aaron Hertzmann. Gaus- sian process dynamical models for human motion. IEEE transactions on pattern analysis and machine intelligence , 30(2):283–298, 2008. 1

  17. [25]

    Spatial tempo- ral graph convolutional networks for skeleton-based action recognition

    Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial tempo- ral graph convolutional networks for skeleton-based action recognition. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. 3

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.