REVIEW 5 major objections 6 minor 33 references
DiVR: incorporating context from diverse VR scenes for human trajectory prediction
T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read DiVR, a cross-modal transformer that represents VR scenes as heterogeneous temporal graphs, predicts human trajectories more accurately than MLP, gaze-only, and point-cloud baselines on the CREATTIVE3D dataset.
desk verdict DiVR is a solid first application of heterogeneous graph context to VR trajectory prediction, but its headline numbers are internally inconsistent and the blanket superiority claim is not supported by all its own tables. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the heterogeneous temporal scene graph, where nodes and edges carry type labels that let the model distinguish a pedestrian waiting at a traffic light from one crossing, and a location from a moving vehicle. A Temporal Graph Convolutional Network with edge-conditioned filters converts the evolving graph into a latent context vector for each timestep; global mean pooling and concatenation produce a spatiotemporal embedding. This context vector is combined in a cross-modal transformer with a PointNet++-encoded gaze-interpolated point cloud and a PerceiverIO motion encoder, so that the final prediction is conditioned on both the raw sensory input and the structured relational description of the environment.
What would settle it
A direct test would be to retrain MLP, TRACK, and GIMO on CREATTIVE3D with the identical data splits, input sampling, and a hyperparameter search (including learning rate, epoch count, and context preprocessing), then measure whether DiVR-Het still beats the best-tuned baseline by the reported margins. A second check would be to replace the heterogeneous graph in DiVR-Het with a homogeneous graph or random edge features: if the ADE/FDE gap over a no-graph version disappears, the improvement is due to the graph structure; if it persists, the gain comes from the architecture or other context.
Extended reading notes
Core claim
The paper's core claim is that representing a virtual urban scene as a directed heterogeneous temporal graph—with typed nodes for locations, pedestrians, vehicles, traffic lights, and buttons, and typed edges for proximity, adjacency, and active interactions—and encoding that graph with a temporal graph convolutional network yields a context feature that, when fused with motion and gaze via a PerceiverIO cross-modal transformer, reduces trajectory prediction error relative to existing baselines. In the balanced CREATTIVE3D evaluation, DiVR-Het reaches an ADE of 0.588 m and an FDE of 0.842 m, versus 0.854/1.512 for the MLP, 0.759/1.247 for TRACK, and 0.757/1.240 for GIMO. The paper also reports that DiVR-Het improves over GIMO on complex-task and low-vision generalization tests when trained on simple tasks, and that graph ablation incurs a larger error increase than gaze ablation, indicating the graph is the more important context modality.
Load-bearing premise
The reported accuracy gains assume that the MLP, TRACK, and GIMO baselines were adapted to the CREATTIVE3D VR setting with equally careful hyperparameter tuning and preprocessing; the experimental section only points to the baselines' original papers and does not state whether they were retrained on the VR data or how their hyperparameters were selected, so DiVR's advantage could partly stem from undertuned comparisons rather than the model itself.
Editorial extensions
If this is right
- DiVR-Het outperforms all three baselines on the primary CREATTIVE3D evaluation, with a 31.2% lower ADE and 44.3% lower FDE than an MLP that sees only past positions.
- Ablation results show that replacing the heterogeneous graph with zeros increases ADE by up to 35.8% under complex tasks, implying the graph contributes more than gaze to prediction accuracy.
- Training on both simple and complex tasks (or on both normal and low vision) improves DiVR's complex-task and low-vision performance beyond the baselines, suggesting the model benefits from exposure to diverse conditions.
- In user generalization, DiVR-Hom achieves the best ADE (0.513) and DiVR-Het the best FDE (0.828) among compared models; task generalization remains difficult, with all models' FDE around 2, but DiVR-Hom still leads in ADE.
- The paper positions heterogeneous graph context as a complement to gaze and point-cloud modalities, and the qualitative examples show DiVR correctly pausing for a traffic light or walking toward a button where GIMO predicts an immediate crossing.
Reading between the lines
- If heterogeneous graph context is indeed the main carrier of the improvement, then a practical route for real-world deployment would be to replace manually annotated VR scene graphs with automatically parsed scene graphs from HD maps and sensor data; a testable extension is to evaluate DiVR on real pedestrian datasets with predicted graphs.
- The ablation gap between graph and gaze suggests that for trajectory prediction, semantic structure (what objects are and how they relate) may matter more than raw perceptual features; a follow-up could test this by feeding the same graph embedding into a simple concatenation model instead of the full cross-modal transformer.
- The mixed results in task generalization hint that the heterogeneous graph's benefit depends on the specific relation types: the model improved in complex tasks involving traffic lights and buttons but not in the symmetric 'return to start' task; an analysis of which edge types drive the gain could guide graph design.
- Because the CREATTIVE3D dataset provides oracle annotations, a natural stress test is to measure how DiVR degrades when the graph is corrupted (e.g., random edges or missing types), which would quantify how much of the gain relies on perfect scene understanding.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DiVR, a cross-modal transformer for human trajectory prediction in VR scenes, built on the PerceiverIO architecture and integrating three modalities: past motion, gaze-interpolated point clouds, and temporal heterogeneous graph context extracted with a TemporalGCN. Using the CREATTIVE3D dataset, the authors report ADE/FDE comparisons against MLP, TRACK, and GIMO baselines, plus ablation studies and generalization tests across users, scenes, and tasks. The central claim is that integrating heterogeneous static and dynamic scene context improves prediction accuracy and adaptability over existing baselines and over homogeneous graphs. Source code is made publicly available.
Significance. If the quantitative claims are validated, the paper would make a useful contribution by demonstrating that high-level scene graphs, rather than raw point clouds, provide effective context for human trajectory prediction, and by extending this line of work to controlled VR environments with simulated low vision. The release of code and the use of a publicly available VR dataset are concrete assets. However, the empirical evidence as presented is not currently anchored: the same model is reported with materially different errors in different tables, baseline adaptation is underspecified, and no uncertainty quantification is provided. The significance of the work therefore depends on the authors reconciling these issues in revision.
major comments (5)
- [§4.2, Tables 3, 4, and 7] The same DiVR-Het configuration is reported with inconsistent ADE/FDE values across tables. Under the named condition NV+ST, Table 4 reports DiVR-Het as 0.604/0.696, Table 7 reports 0.535/0.705, and Table 3 reports 0.588/0.842 without naming the condition; the Table 3 numbers are the basis for the 31.2% and 44.3% reductions stated in §4.2. Similarly, the complex-task row for DiVR-Het is 0.910/1.661 in Table 4 but 0.695/1.218 in Table 7. Baselines also differ across tables (e.g., MLP is 0.854/1.512 in Table 3 and 0.652/1.139 in Table 4). The paper must state which protocol, data split, and training condition each table refers to, and the headline claim must be tied to a single reproducible evaluation protocol.
- [§4.1] The adaptation of baselines to the VR setting is not specified. The paper cites original papers for MLP, TRACK, and GIMO but does not state whether they were retrained on CREATTIVE3D, how their hyperparameters were chosen, or how GIMO's gaze and point-cloud pipeline was adapted from real-world motion capture to the 2D head-position VR data used here. If the baselines were not comparably tuned or preprocessed, DiVR's reported gains could be an artifact of the comparison rather than a real modeling improvement. Please report the exact adaptation protocol, hyperparameter selection, and tuning budget for each baseline.
- [§4.2, Tables 4 and 6] The abstract's blanket claim that DiVR achieves higher accuracy than other models is not supported by several reported rows. In Table 4, under NV+ST, GIMO and TRACK achieve lower ADE (0.509 and 0.515) than DiVR-Het (0.604). In Table 6, under scene generalization GIMO has a lower ADE (0.997 vs 1.073), and under task generalization GIMO has both a lower ADE (1.296 vs 1.384) and a lower FDE (2.100 vs 2.605) than DiVR-Het. The superiority claims should be qualified to the conditions and metrics where DiVR is actually better, or the numerical results should be corrected.
- [All tables and §4.2] No error bars, confidence intervals, or significance tests are reported. All tables present single-run point estimates, and the user-generalization experiment is based on one random selection of 10 test users. Many of the differences used to support the central claim are small (e.g., Table 4 low-vision ADE: DiVR-Het 0.615 vs GIMO 0.630 vs TRACK 0.636), so without repeated runs or resampled splits it is impossible to distinguish real gains from noise. Please report results over multiple seeds and/or user-split resamplings with standard deviations and a significance test.
- [§4.1 and §4.2] It is not stated whether the 70/15/15 train/validation/test split is user-disjoint. Section 4.2 defines user generalization separately by holding out 10 users, which suggests the standard context-evaluation split may mix trajectories from the same user across train and test. If so, the reported accuracy and generalization numbers would be inflated. The paper must clarify the split granularity and, if the goal is generalization, ensure that the standard split is also user-disjoint.
minor comments (6)
- [§2] In the Graph-based techniques paragraph, 'trajectory precition' should be 'trajectory prediction'.
- [Figure 4 caption] The caption writes 'DiVT-het' where the model name should be 'DiVR-Het'.
- [§3.2] The loss function components Ltrans, Lrec, and Ldes_trans are named but not defined; please provide exact equations and specify the relative weights used in training.
- [§3.1] The rules for assigning edge types (approach, adjacent, interaction) and their temporal attributes from the CREATTIVE3D annotations should be stated precisely, as they determine both the graph representation and the input to the TemporalGCN.
- [§4.2, Table 5 discussion] The sentence 'DiVR-Hom reduces its ADE from 0.592 to 0.470 and its FDE from 0.908 to 0.816' compares results under different test conditions (NV+ST vs Complex Task) rather than the same condition under different training regimes; please report the intended like-for-like comparison.
- [§4.1] Since the Perceiver architecture is motivated by efficiency, reporting model parameter counts and training/inference time would strengthen the presentation.
Circularity Check
No circular derivation: DiVR's accuracy claim rests on an empirical comparison against external baselines, not on fitted inputs or load-bearing self-citations.
full rationale
The central claim of the paper is empirical: DiVR achieves lower ADE/FDE than MLP, TRACK, and GIMO on the CREATTIVE3D dataset. The model is trained end-to-end with a supervised loss (L_total = L_trans + L_rec + L_des_trans) on a 70/15/15 split, and the reported metrics are computed on held-out test trajectories. No equation or construction defines the predicted trajectory in terms of the evaluation metric, and the headline percentage improvements are not enforced by the model definition. The heterogeneous graph representation uses scene annotations and current interaction states, not future ground-truth positions, so it does not smuggle the target into the input. The paper does cite the authors' own dataset [22,23] and the prior benchmark [6], but these are provenance citations and do not supply the numerical superiority claim, which is established by comparison with external architectures GIMO, TRACK, and MLP. The stated limitation in the conclusion, that the model relies on high-quality datasets for scene graph creation, is a scope limitation rather than evidence of circularity. The numerical inconsistencies between Tables 4 and 7 under the same named condition are a reproducibility and correctness concern, not a circular-reasoning concern, because they do not reduce the claim to its own inputs. No self-definitional step, fitted-input-as-prediction, or self-citation chain carrying the central result was identified.
Assumptions & free parameters
free parameters (9)
- Latent dimension =
256
- Number of encoding layers =
6
- Number of attention heads =
8
- Learning rate =
0.0001
- Weight decay =
0.0001
- Batch size =
16
- Training epochs =
100
- Observation window =
3 seconds at 2 fps
- Prediction horizon =
5 seconds
assumptions (4)
- domain assumption PerceiverIO cross-attention and TemporalGCN are effective inductive biases for trajectory prediction from graphs, point clouds, and gaze.
- domain assumption The CREATTIVE3D annotations (node types, edge types, interaction attributes) accurately represent the scene semantics that determine pedestrian motion.
- domain assumption A 2D head position in the 10x4m tracked space is a sufficient state representation for the trajectory prediction task.
- standard math Standard stochastic gradient training with Adam converges to a good local optimum for all compared models.
Cite this review
Pith. "Pith review of DiVR: incorporating context from diverse VR scenes for human trajectory prediction." pith.science (2026). https://pith.science/paper/JGRCWFNK
@misc{pith2026241108409,
author = {Pith},
title = {Pith review of: DiVR: incorporating context from diverse VR scenes for human trajectory prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/JGRCWFNK}},
note = {Machine review of arXiv:2411.08409}
}
read the original abstract
Virtual environments provide a rich and controlled setting for collecting detailed data on human behavior, offering unique opportunities for predicting human trajectories in dynamic scenes. However, most existing approaches have overlooked the potential of these environments, focusing instead on static contexts without considering userspecific factors. Employing the CREATTIVE3D dataset, our work models trajectories recorded in virtual reality (VR) scenes for diverse situations including road-crossing tasks with user interactions and simulated visual impairments. We propose Diverse Context VR Human Motion Prediction (DiVR), a cross-modal transformer based on the Perceiver architecture that integrates both static and dynamic scene context using a heterogeneous graph convolution network. We conduct extensive experiments comparing DiVR against existing architectures including MLP, LSTM, and transformers with gaze and point cloud context. Additionally, we also stress test our model's generalizability across different users, tasks, and scenes. Results show that DiVR achieves higher accuracy and adaptability compared to other models and to static graphs. This work highlights the advantages of using VR datasets for context-aware human trajectory modeling, with potential applications in enhancing user experiences in the metaverse. Our source code is publicly available at https://gitlab.inria.fr/ffrancog/creattive3d-divr-model.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
In: 2021 International Conference on 3D Vision (3DV)
Aksan, E., Kaufmann, M., Cao, P., Hilliges, O.: A spatio-temporal transformer for 3d human motion prediction. In: 2021 International Conference on 3D Vision (3DV). pp. 565–574. IEEE (2021)
work page 2021
-
[2]
Transportation Research Part C: Emerging Technologies157, 104405 (2023)
Carrasco Limeros, S., Majchrowska, S., Johnander, J., Petersson, C., Fernández Llorca, D.: Towards explainable motion prediction using heterogeneous graph rep- resentations. Transportation Research Part C: Emerging Technologies157, 104405 (2023)
work page 2023
-
[3]
IEEE Transactions on Neural Networks and Learning Systems (2021)
Chen, J., Wang, Y., Wu, R., Campbell, M.: Spatial temporal graph neural network for interaction-aware vehicle trajectory prediction. IEEE Transactions on Neural Networks and Learning Systems (2021)
work page 2021
-
[4]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Corona, E., Pumarola, A., Alenya, G., Moreno-Noguer, F.: Context-aware human motion prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6992–7001 (2020)
work page 2020
-
[5]
Advances in neural information processing systems28 (2015)
Duvenaud,D.K.,Maclaurin,D.,Iparraguirre,J.,Bombarell,R.,Hirzel,T.,Aspuru- Guzik, A., Adams, R.P.: Convolutional networks on graphs for learning molecular fingerprints. Advances in neural information processing systems28 (2015)
work page 2015
-
[6]
Gallo, F.F., Wu, H.Y., Sassatelli, L.: Human trajectory forecasting in 3d envi- ronments: Navigating complexity under low vision. In: Proceedings of the 16th International Workshop on Immersive Mixed and Virtual Environment Systems. pp. 57–63 (2024)
work page 2024
-
[7]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Gao, K., Chen, L., Niu, Y., Shao, J., Xiao, J.: Classification-then-grounding: Re- formulating video scene graphs as temporal bipartite graphs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19497– 19506 (2022)
work page 2022
-
[8]
In: Proceedings of the 34th International Confer- ence on Machine Learning - Volume 70
Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E.: Neural message passing for quantum chemistry. In: Proceedings of the 34th International Confer- ence on Machine Learning - Volume 70. p. 1263–1272. ICML’17, JMLR.org (2017)
2017
Show all 33 references
-
[9]
In: Proc
Guo, W., Du, Y., Shen, X., Lepetit, V., Alameda-Pineda, X., Moreno-Noguer, F.: Back to mlp: A simple baseline for human motion prediction. In: Proc. IEEE Winter Conf. on Appl. of Computer Vision. pp. 4809–19 (2023)
2023
-
[10]
arXiv preprint arXiv:2107.14795 (2021)
Jaegle,A.,Borgeaud,S.,Alayrac,J.B.,Doersch,C.,Ionescu,C.,Ding,D.,Koppula, S., Zoran, D., Brock, A., Shelhamer, E., et al.: Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795 (2021)
2021 arXiv
-
[11]
Journal of Advanced Computer Vision (2021)
Jones, R., Lee, D.: Challenges in 3d point cloud processing for urban scene under- standing. Journal of Advanced Computer Vision (2021)
2021
-
[12]
Journal of computer-aided molecular design 30, 595–608 (2016)
Kearnes, S., McCloskey, K., Berndl, M., Pande, V., Riley, P.: Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design 30, 595–608 (2016)
2016
-
[13]
arXiv preprint arXiv:2109.00181 (2021)
Li, H., et al.: Ctal: Pre-training cross-modal transformer for audio-and-language representations. arXiv preprint arXiv:2109.00181 (2021)
2021 arXiv
-
[14]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Li, W., Mahadevan, V., Vasconcelos, N.: Situation recognition: Visual semantic role labeling for image understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
2017
-
[15]
In: Proceedings of the International Conference on Learning Representations (2020)
Luo, S., et al.: C4av: learning cross-modal representations from transformers. In: Proceedings of the International Conference on Learning Representations (2020)
2020
-
[16]
IEEE Transactions on Pattern Analysis and Machine Intelligence 44(9), 5681–5699 (2021) 16 F.Franco et al
Rondón, M.F.R., Sassatelli, L., Aparicio-Pardo, R., Precioso, F.: Track: A new method from a re-examination of deep architectures for head motion prediction in 360o videos. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(9), 5681–5699 (2021) 16 F.Franco et al
2021
-
[17]
arXiv preprint arXiv:2001.03093 (2021)
Salzmann, T., Ivanovic, B., Chakravarty, P., Pavone, M.: Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. arXiv preprint arXiv:2001.03093 (2021)
2021 arXiv
-
[18]
In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Simonovsky, M., Komodakis, N.: Dynamic edge-conditioned filters in convolutional neural networks on graphs. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 29–38 (2017)
2017
-
[19]
Journal of Security and Safety Technologies (2020)
Smith, A., et al.: Limitations of surveillance cameras in public spaces: A case study. Journal of Security and Safety Technologies (2020)
2020
-
[20]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops
Sun, R., Lingrand, D., Precioso, F.: Exploring the road graph in trajectory fore- casting for autonomous driving. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops. pp. 71–80 (October 2023)
2023
-
[21]
In: Advances in neural information processing systems
Vaswani, A., et al.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017)
2017
-
[22]
Wu, H.Y., Robert, F.A.S., Gallo, F.F., Pirkovets, K., Quere, C., Delachambre, J., Ramanoël, S., Gros, A., Winckler, M., Sassatelli, L., Hayotte, M., Menin, A., Ko- rnprobst, P.: Exploring, walking, and interacting in virtual reality with simulated low vision: a living contextu...
2023
-
[23]
Wu, H.Y., Robert, F.A.S., Gallo, F.F., Pirkovets, K., Quere, C., Delachambre, J., Ramanoël, S., Gros, A., Winckler, M., Sassatelli, L., et al.: Exploring, walking, and interacting in virtual reality with simulated low vision: a living contextual dataset (2023)
2023
-
[24]
In: AAAI (2018)
Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: AAAI (2018)
2018
-
[25]
In: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction
Yasar, M.S., Islam, M.M., Iqbal, T.: Posetron: Enabling close-proximity human- robot collaboration through multi-human motion prediction. In: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction. pp. 830–839 (2024)
2024
-
[26]
In: European Conference on Computer Vision – ECCV 2020
Yu, C., Ma, X., Ren, J., Zhao, H., Yi, S.: Spatio-temporal graph transformer net- works for pedestrian trajectory prediction. In: European Conference on Computer Vision – ECCV 2020. pp. 507–523 (2020)
2020
-
[27]
In: Proceedings of the British Machine Vision Conference (BMVC) (2023)
Zaier, M., Wannous, H., Drira, H., Boonaert, J.: Cross-modal attention for accurate pedestrian trajectory prediction. In: Proceedings of the British Machine Vision Conference (BMVC) (2023)
2023
-
[28]
CIRP Annals (2020)
Zhang, J., Liu, H., Chang, Q., Wang, L., Gao, R.: Recurrent neural network for motion trajectory prediction in human-robot collaborative assembly. CIRP Annals (2020)
2020
-
[29]
on Computer Vision and Pattern Recognition (CVPR)
Zhao, T., Xu, Y., Monfort, M., Choi, W., Baker, C.L., Zhao, Y., Wang, Y., Wu, Y.N.:Multi-agenttensorfusionforcontextualtrajectoryprediction.In:IEEEConf. on Computer Vision and Pattern Recognition (CVPR). pp. 12126–12134 (2019)
2019
-
[30]
In: European Conference on Computer Vision (2022)
Zheng, Y., Yang, Y., Mo, K., Li, J., Yu, T., Liu, Y., Liu, K., Guibas, L.J.: Gimo: Gaze-informed human motion prediction in context. In: European Conference on Computer Vision (2022)
2022
-
[31]
Knowledge-Based Sys- tems 277, 110775 (2023)
Zhou, H., Yang, X., Fan, M., Huang, H., Ren, D., Xia, H.: Static-dynamic global graph representation for pedestrian trajectory prediction. Knowledge-Based Sys- tems 277, 110775 (2023)
2023
-
[32]
AI Open1, 57–81 (2020) DiVR: Incorporating context from VR for Human Trajectory Prediction 17
Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., Sun, M.: Graph neural networks: A review of methods and applications. AI Open1, 57–81 (2020) DiVR: Incorporating context from VR for Human Trajectory Prediction 17
2020
-
[33]
Journal of Traffic and Transportation Engineering (2023)
Zipfl, M., Hertlein, F., Rettinger, A., Thoma, S., Halilaj, L., Luettin, J., Schmid, S., Henson, C.: Relation-based motion prediction using traffic scene graphs. Journal of Traffic and Transportation Engineering (2023)
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.