Pith. sign in

REVIEW 3 major objections 6 minor 45 references

Compositional Video Prediction

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that predicting future frames by modeling scene entities and a single trajectory-level latent variable yields realistic stochastic video prediction from one image.

desk verdict A genuinely useful compositional architecture for stochastic video prediction, but the headline claim about the global latent runs ahead of the best-of-100 evidence. read the letter →

arxiv 1908.08522 v1 pith:LV4STQXS submitted 2019-08-22 cs.CV

Yufei Ye , Maneesh Singh , Abhinav Gupta , Shubham Tulsiani This is my paper · ORCID
classification cs.CV
keywords videopredictionstochasticgenerationentityfactorizationgraphneuralnetworkslatentvariablefutureframesynthesisscenecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that future frames should be predicted not by transforming pixels wholesale, but by first decomposing a scene into entities, forecasting each entity's location and appearance, and then composing the next frame from those forecasts while allowing interactions between entities. It further claims that the inherent ambiguity of the future is best handled by a single global latent variable representing the whole trajectory, which then generates per-timestep conditioning; this produces both more realistic frames and a wider spread of plausible futures than independent per-timestep randomness. The approach is validated on falling-block towers and gym-sport videos, where it outperforms frame-level baselines and per-timestep stochastic models on both entity-location error and perceptual frame distance. If the claim holds, video prediction shifts from pixel statistics to object- and interaction-level reasoning, making forecasts interpretable and generalizable to novel scene configurations.

What carries the argument

The central mechanism is the pair of factored representations and the global trajectory latent. Each entity is a tuple of a 2D location and a 32-dimensional appearance vector; the predictor evolves these tuples through four rounds of message passing in a graph (fully connected for objects, a skeleton for human joints). A one-layer LSTM converts a single sampled latent $u$ into per-timestep conditions $z_t$, so all timesteps are correlated through one draw. The decoder decodes a normalized spatial feature and a soft mask per entity, warps both to image coordinates via a Spatial Transformer Network, and composites them over background features by weighted averaging, resolving overlap with the masks. This machinery is what allows the model to reason about interactions, generalize to different numbers of entities, and sample coherent futures.

What would settle it

Run the trained model on a video of a single block rotating about its center with no translation: if the appearance feature is orientation-blind, the composed frames will blur or smear the block's faces. A complementary check on diversity: record the 100 sampled futures without cherry-picking; if the set collapses to a single trajectory, the global latent carries no ambiguity, and if the best-of-100 score is high while the average is near-identical across samples, the evaluation protocol hides the collapse.

Watch

Extended reading notes

Core claim

The central claim is that a scene can be factorized into entities—each represented as a 2D location and a learned appearance feature—and that predicting forward in this factored space, then decoding back to pixels, gives better stochastic video prediction than predicting frames directly. The predictor is a graph neural network that passes messages between entities so that motion depends on the configuration of neighbors (for example, whether a block will fall depends on the blocks around it). The decoder warps per-entity appearance features and soft visibility masks to their predicted locations and blends them over background features extracted from the initial frame, which lets it handle occlusions and newly visible background. Ambiguity is resolved by a global latent variable that is drawn once per video and expanded through an LSTM into per-timestep conditions, so that all frames of a sample share one coherent future. The paper shows on ShapeStacks and Penn Action that this composition outperforms alternatives and that the global latent produces consistent yet diverse sample trajectories.

Load-bearing premise

The method assumes every scene can be fully captured by a fixed set of entities, each described by a 2D location and a 32-dimensional appearance vector, and that composing decoded features from these entities over the initial frame reconstructs a realistic future frame — so any appearance change the feature cannot encode (orientation, deformation, self-occlusion) degrades the composition.

Editorial extensions

If this is right

  • A single global latent draw produces trajectories that stay consistent across time, whereas independent per-timestep sampling drifts or switches between futures partway through a clip.
  • Entity-level factorization lets the model predict scenes with more entities than seen in training; the paper shows reasonable generalization from 3-block training to 4-, 5-, and 6-block test towers.
  • The learned decoder's soft masks act as unsupervised instance segmentations, which could be reused for other tasks such as object discovery or scene editing.
  • Because entity locations are the only supervision the predictor needs per frame, replacing manual annotations with an off-the-shelf pose detector at both train and test time yields competitive results, suggesting the method tolerates noisy entity localization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension, unstated by the paper, is to condition or edit the global latent $u$ directly to steer the future—choosing the direction a tower topples or the activity a person performs—turning the predictor into a controllable video generator.
  • The appearance feature is 32-dimensional and orientation-blind; a stress test with objects that rotate significantly (rather than merely toppling) would reveal whether the decoder or the feature must take on pose information.
  • The best-of-100 evaluation rewards coverage rather than diversity; measuring the spread or calibration of samples without cherry-picking would test whether the latent's diversity is usable.
  • The decomposition into entities is assumed given; the model itself does not discover entities. Marrying this predictor with an unsupervised entity-discovery module would point toward prediction without any location supervision.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a compositional approach to stochastic video prediction from a single annotated image. The scene is decomposed into entities, each represented by a 2D location and a 32-dimensional appearance feature. A graph neural network predictor propagates these representations through time, conditioned on per-timestep latent variables z_t that are derived deterministically from a single global trajectory-level latent u via an LSTM. A learned decoder warps per-entity appearance features and soft masks to the predicted locations and composes them with background features to render future frames. The model is trained by maximizing a variational lower bound with an auxiliary location loss and an auto-encoding decoder loss. Experiments on ShapeStacks and Penn Action compare the approach against frame-level, non-interaction, per-timestep-latent, and pose-based baselines, reporting location error and LPIPS under a best-of-100 sampling protocol.

Significance. If the empirical claims are confirmed, the paper makes a useful contribution: it demonstrates that explicit entity factorization with a global trajectory-level latent can yield competitive stochastic video prediction on both synthetic and real domains, and the decoder design (late fusion of warped entity features with soft masks) is a clean, reusable component. The paper deserves credit for testing multiple baselines and ablations, for examining generalization to different numbers of entities, and for showing robustness to detected rather than annotated keypoints. The main weakness is that the evaluation protocol (best-of-100 selection) and the train-with-ground-truth-latent decoder ablation do not fully support the paper's central claims about realism and the advantage of the global latent over per-timestep latents.

major comments (3)
  1. [Section 4.1, Evaluation Metrics; Figures 5, 10, 14] The central claim that the global trajectory-level latent produces 'more realistic predictions' than per-timestep latents (Section 1) rests on best-of-100 curves and on qualitative examples selected as the best LPIPS sample. Best-of-N error is a lower envelope and does not measure the quality of a typical or randomly drawn sample; a model whose posterior is broad but mostly poor can score well on best-of-100. The paper does not report mean or median errors over the 100 samples in the main comparisons (Figure 15 reports sigma only for ShapeStacks location error and only for the proposed model). Please report average/median sample quality (or another proper stochastic metric such as a diversity-accuracy curve) for all methods, and state whether the qualitative 'best' selections are representative of typical outputs.
  2. [Section 4.2, Primitive Decoder] The decoder ablation is performed with 'the latent u encoded by the ground truth videos,' i.e., conditioned on the actual future. This protocol turns the decoder comparison into a reconstruction/auto-encoding test rather than a prediction test, so the conclusion that late fusion is the best composition strategy for generating future frames is not supported by the presented evidence. Please re-run the decoder comparison under the inference-time protocol (sampled u from the prior, predicted entity representations) or explicitly justify why conditioning on the ground truth does not change the relative ordering.
  3. [Section 4, Figures 5-14] Quantitative comparisons are presented as curves without error bars or significance tests over the test set or over random seeds. Given that the differences between the proposed model and baselines (e.g., Figures 10 and 14) appear modest at several time steps, error bars are needed to establish that the reported advantages are not within noise. Additionally, the claim in Section 4.3 that using detected keypoints is 'competitive' with annotated keypoints is only supported qualitatively (Figure 12); please provide the quantitative comparison.
minor comments (6)
  1. [Appendix A, Entity Predictor] In the e->v message passing step, the equation reads 'v^{(l+1)}_i = f^{(l)}_{v->e}[POOL[e^{(l)}_{i,j}|(i,j)]]', but the function should likely be f^{(l)}_{e->v}; please fix this typo.
  2. [Section 3.1, Entity Predictor] After the final interaction block, the text says 'the final block outputs predictions for the entity features for the next timestep {x_t_n}N_n=1', but the notation should be {x^{t+1}_n} to be consistent with the surrounding equations.
  3. [Section 3.2, Eq. (2)] The symbol '⊕' is used in the composition equation without being defined; it appears to denote elementwise addition, and should be stated explicitly.
  4. [Section 3.4, Training Objective] The sentence 'We will release our code for reproducibility' is a promise that is not fulfilled in the submitted manuscript; either provide a code link or soften the statement.
  5. [Figure 15 / Appendix D] Figure 15 is only discussed in Appendix D; its sigma curves would be more informative if referenced in Section 4.2 when discussing sample diversity, and ideally for all compared methods.
  6. [Section 4.2, Figure 6] The qualitative discussion references 'later time steps' for the No-Factor baseline, but the time steps are not clearly annotated in Figure 6; consider labeling the visualized frames with the corresponding time indices.

Circularity Check

1 steps flagged · score 2.0 of 10

No central circularity: the main comparison samples from the prior and matches baselines under the same protocol; the only self-referential step is the decoder ablation, which visibly conditions on the ground-truth future latent.

  1. fitted input called prediction [Sec. 4.1 Evaluation Metrics; Sec. 4.2 Primitive Decoder]
    "When we ablate non-stochastic modules (e.g. decoders), we use the mean u predicted by the latent encoder (after seeing the 'ground-truth' video). ... Note the latent u is encoded by the ground truth videos."

    In the primitive-decoder ablation, the latent u is produced by the trained encoder from the ground-truth future (q(u|f0, f̂T)), and the decoder is trained with Ldec to reconstruct exactly that future from ground-truth entity features. Evaluating decoder variants with this u therefore measures autoencoding/reconstruction capacity with an oracle summary of the target, not single-image future prediction. The paper is transparent about this, and the step is used only to select the decoder fusion level; the central stochastic-prediction claim is not reduced by this choice.

full rationale

The paper makes no formal derivation that reduces to its inputs; it is an empirical architecture and representation paper. The central comparison (global trajectory-level latent vs per-timestep latents) is evaluated by sampling 100 latents from the prior for all models and recording best errors, so the proposed model does not receive the ground-truth future at inference in the main experiments. Best-of-100 evaluation is a limitation for claims about typical-sample realism and diversity, but it is not circular because the samples are drawn from the model prior and the same protocol is applied to baselines. The only self-referential element is the decoder ablation, which uses the latent encoder's mean conditioned on the ground-truth video; the paper explicitly flags this ('Note the latent u is encoded by the ground truth videos') and uses it only to compare fusion architectures, not to establish the headline prediction claim. Self-citations to prior work involving co-authors (e.g., Pose-Knows) are used as baselines, not as load-bearing justification. Accordingly, no significant circularity is present beyond the limited decoder-ablation step, and the appropriate score is 2.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper's central claim rests on standard deep learning components, hand-tuned hyperparameters, and domain assumptions about entity representation, graph structure, and evaluation metrics. No new physical or mathematical entities are introduced. The training objective is a heuristic surrogate for ELBO, and the evaluation protocol has known limitations.

free parameters (5)
  • lambda_1 = 100
    Weight for location supervision in the prediction loss; chosen by hand and used in all experiments.
  • lambda_2 = 1e-3
    Weight for the KL divergence term; chosen by hand and used in all experiments.
  • latent_dim = 8
    Dimensionality of the global latent variable u and per-timestep variables z_t; chosen by hand.
  • appearance_dim = 32
    Dimensionality of per-entity appearance features a_n; chosen by hand and used throughout.
  • entity_region_size = 70
    Fixed width and height of the region used to extract entity appearance features; chosen by hand.
assumptions (5)
  • domain assumption The training objective in Section 3.4 is treated as a valid variational lower bound on the log-likelihood of the future frames, even though the implemented loss is a weighted sum of reconstruction, autoencoding, and KL terms, and the decoder output is not a normalized likelihood.
    The paper uses the ELBO formalism, but the actual loss combines L1 frame loss, L2 location loss, an auxiliary autoencoding loss, and a KL term. This is common in practice but is not a formal ELBO, so the likelihood interpretation is approximate.
  • domain assumption At test time, entity locations are available either from ground-truth annotations or from an off-the-shelf detector, and the model is robust to noise in these locations.
    Section 4.1 says the authors 'substitute ground truth annotation with key-points location from off-the-shelf detector in both training and testing', and Section 5 states the reliance on entity locations as a limitation. The central method depends on this availability.
  • domain assumption A fixed-size appearance vector (32 dims) and a 2D location are sufficient to represent an entity's state, and the graph predictor can propagate this state over time, including rotations, deformations, and interactions.
    Sections 3.1 and 3.2; if appearance cannot change under viewpoint or deformation, the composed frames will be unrealistic. The experiments show this works for two domains, but it is an assumption about representational sufficiency.
  • domain assumption The graph structure used for message passing, fully connected for ShapeStacks and skeleton for Penn Action, accurately reflects the interaction patterns among entities.
    Section 3.1 and Appendix A. For Penn Action, some joints may be missing; edges are relinked to parents (Appendix B). If the graph were wrong, interactions would be mis-modeled.
  • domain assumption The evaluation metrics, LPIPS and location MSE with best-of-100 sampling, are valid proxies for video prediction quality.
    Section 4.1 describes the metrics. Best-of-100 is known to favor high-variance stochastic models, and LPIPS is a perceptual metric, but the paper does not justify why these are the right metrics for the claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Compositional Video Prediction." pith.science (2026). https://pith.science/paper/LV4STQXS

@misc{pith2026190808522,
  author       = {Pith},
  title        = {Pith review of: Compositional Video Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LV4STQXS}},
  note         = {Machine review of arXiv:1908.08522}
}
read the original abstract

We present an approach for pixel-level future prediction given an input image of a scene. We observe that a scene is comprised of distinct entities that undergo motion and present an approach that operationalizes this insight. We implicitly predict future states of independent entities while reasoning about their interactions, and compose future video frames using these predicted states. We overcome the inherent multi-modality of the task using a global trajectory-level latent random variable, and show that this allows us to sample diverse and plausible futures. We empirically validate our approach against alternate representations and ways of incorporating multi-modality. We examine two datasets, one comprising of stacked objects that may fall, and the other containing videos of humans performing activities in a gym, and show that our approach allows realistic stochastic video prediction across these diverse settings. See https://judyye.github.io/CVP/ for video predictions.

Figures

Figures reproduced from arXiv: 1908.08522 by the authors.

Figure 1
Figure 1. Given a still image with locations of entities (objects or [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our model takes as input an image with known/detected location of entities. Each entity is represented as its location and an [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Our encoder (a) and baseline encoder (b-d). At test time, [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Error for location (Left) and frame (Right) prediction [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Video predictions using our predictor compared to baselines. We visualize the generated sequence after every 3 time steps. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Above: Quantitative evaluation of the entity predictor [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 10
Figure 10. Figure 10: Error for location (Left) and frame (Right) prediction [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 9
Figure 9. Figure 9: Left: Average Perceptual error for predicted frames via [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 12
Figure 12. Figure 12: Video prediction results with best LPIPS latent using our approach compared to baselines. The last column visualizes results [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: Visualizing joint positions in three randomly sampled [PITH_FULL_IMAGE:figures/full_fig_p008_13.png]
Figure 14
Figure 14. Figure 14: Error for location (Left) and frame (Right) prediction [PITH_FULL_IMAGE:figures/full_fig_p008_14.png]
Figure 15
Figure 15. Figure 15: Left: For all 100 samples, σ (shade) and best samples (line at lower boundary of shade) are plotted. Right: For best 5 samples of 100, mean and σ are plotted. D. Standard Deviation Prediction task is multi-modal, a model that correctly handles uncertainty will predict…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 39 canonical work pages

  1. [1]

    Stochastic variational video prediction

    Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. ICLR, 2017. 2

    work page 2017

  2. [2]

    Interaction networks for learning about objects, relations and physics

    Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In NeurIPS, 2016. 2, 3

    work page 2016

  3. [3]

    Relational inductive biases, deep learn- ing, and graph networks

    Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Al- varo Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Ma- linowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learn- ing, and graph networks. arXiv preprint arXiv:1806.01261 ,

    arXiv

  4. [4]

    A compositional object-based ap- proach to learning physical dynamics

    Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based ap- proach to learning physical dynamics. ICLR, 2016. 2, 8

    work page 2016

  5. [5]

    Video imagination from a single image with transformation gener- ation

    Baoyang Chen, Wenmin Wang, and Jinzhuo Wang. Video imagination from a single image with transformation gener- ation. In ACMMM Workshop, 2017. 2

    work page 2017

  6. [6]

    Stochastic video generation with a learned prior

    Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In ICML, 2018. 2, 5, 7

    work page 2018

  7. [7]

    Unsupervised learning of disentangled representations from video

    Emily L Denton et al. Unsupervised learning of disentangled representations from video. In NeurIPS, 2017. 2

    work page 2017

  8. [8]

    RMPE: Regional multi-person pose estimation

    Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. RMPE: Regional multi-person pose estimation. In ICCV,

    work page

Show all 45 references
  1. [9]

    Unsuper- vised learning for physical interaction through video predic- tion

    Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsuper- vised learning for physical interaction through video predic- tion. In NeurIPS, 2016. 2

    2016

  2. [10]

    Learning visual predictive models of physics for playing billiards

    Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. ICLR, 2015. 2

    2015

  3. [11]

    Motion prediction under multimodality with conditional stochastic networks

    Katerina Fragkiadaki, Jonathan Huang, Alex Alemi, Sud- heendra Vijayanarasimhan, Susanna Ricco, and Rahul Sukthankar. Motion prediction under multimodality with conditional stochastic networks. arXiv preprint arXiv:1705.02082, 2017. 2

    2017 arXiv

  4. [12]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS,

  5. [13]

    Shapestacks: Learning vision-based physical in- tuition for generalised object stacking

    Oliver Groth, Fabian Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision-based physical in- tuition for generalised object stacking. ECCV, 2018. 2, 5

    2018

  6. [14]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,

  7. [15]

    Learning to decompose and disentangle representations for video prediction

    Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. NeurIPS, 2018. 2, 8

    2018

  8. [16]

    Approximate maxent inverse optimal control and its application for mental simulation of human interactions

    De-An Huang, Amir-massoud Farahmand, Kris M Kitani, and James Andrew Bagnell. Approximate maxent inverse optimal control and its application for mental simulation of human interactions. In AAAI, 2015. 2

    2015

  9. [17]

    Spatial transformer networks

    Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NeurIPS, 2015. 4

    2015

  10. [18]

    Dynamic filter networks

    Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. In NeurIPS, 2016. 2

    2016

  11. [19]

    3d reasoning from blocks to stability

    Zhaoyin Jia, Andrew C Gallagher, Ashutosh Saxena, and Tsuhan Chen. 3d reasoning from blocks to stability. IEEE transactions on pattern analysis and machine intelligence ,

  12. [20]

    Auto-encoding varia- tional bayes

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. ICLR, 2014. 2, 3

    2014

  13. [21]

    Neural relational inference for interacting systems

    Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. ICML, 2019. 2, 3

    2019

  14. [22]

    Activity forecasting

    Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting. In ECCV, 2012. 2

    2012

  15. [23]

    Learning physical intuition of block towers by example

    Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. ICML, 2016. 2, 5, 6

    2016

  16. [24]

    To fall or not to fall: A visual approach to physical stability prediction

    Wenbin Li, Seyedmajid Azimi, Ale ˇs Leonardis, and Mario Fritz. To fall or not to fall: A visual approach to physical stability prediction. arXiv preprint arXiv:1604.00066, 2016. 2

    2016 arXiv

  17. [25]

    Visual stability prediction and its application to manipulation

    Wenbin Li, Ale ˇs Leonardis, and Mario Fritz. Visual stability prediction and its application to manipulation. AAAI, 2016. 2

    2016

  18. [26]

    Video frame synthesis using deep voxel flow

    Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In ICCV, 2017. 2

    2017

  19. [27]

    Deep multi-scale video prediction beyond mean square error

    Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. ICLR, 2016. 2

    2016

  20. [28]

    A simple neural network module for relational reason- ing

    Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lilli- crap. A simple neural network module for relational reason- ing. In NeurIPS, 2017. 2

    2017

  21. [29]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012. 11

    2012

  22. [30]

    Learning spatiotemporal features with 3d convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015. 8, 11

    2015

  23. [31]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. CVPR, 2017. 2

    2017

  24. [32]

    Decomposing motion and content for nat- ural video sequence prediction

    Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for nat- ural video sequence prediction. ICLR, 2017. 2

    2017

  25. [33]

    Learning to generate long- term future via hierarchical prediction

    Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long- term future via hierarchical prediction. ICML, 2017. 2

    2017

  26. [34]

    Generating videos with scene dynamics

    Carl V ondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NeurIPS, 2016. 2

    2016

  27. [35]

    An uncertain future: Forecasting from static images using variational autoencoders

    Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In ECCV, 2016. 2

    2016

  28. [36]

    The pose knows: Video forecasting by generating pose futures

    Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Video forecasting by generating pose futures. In ICCV, 2017. 2, 5, 8, 11

    2017

  29. [37]

    High-resolution image syn- thesis and semantic manipulation with conditional gans

    Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image syn- thesis and semantic manipulation with conditional gans. In CVPR, 2018. 11

    2018

  30. [38]

    Visual in- teraction networks

    Nicholas Watters, Andrea Tacchetti, Theophane Weber, Raz- van Pascanu, Peter Battaglia, and Daniel Zoran. Visual in- teraction networks. arXiv preprint arXiv:1706.01433, 2017. 2

    2017 arXiv

  31. [39]

    Physics 101: Learning physi- cal object properties from unlabeled videos

    Jiajun Wu, Joseph J Lim, Hongyi Zhang, Joshua B Tenen- baum, and William T Freeman. Physics 101: Learning physi- cal object properties from unlabeled videos. InBMVC, 2016. 2

    2016

  32. [40]

    Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks

    Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Free- man. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In NeurIPS, 2016. 2

    2016

  33. [41]

    MT-V AE: learning motion transformations to generate multimodal human dynamics

    Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap, Ersin Yumer, and Honglak Lee. MT-V AE: learning motion transformations to generate multimodal human dynamics. In ECCV, 2018. 2

    2018

  34. [42]

    Interpretable intuitive physics model

    Tian Ye, Xiaolong Wang, James Davidson, and Abhinav Gupta. Interpretable intuitive physics model. ECCV, 2018. 2

    2018

  35. [43]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 5

    2018

  36. [44]

    From actemes to action: A strongly-supervised repre- sentation for detailed action understanding

    Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpa- nis. From actemes to action: A strongly-supervised repre- sentation for detailed action understanding. In ICCV, 2013. 5 A. Architecture Details Entity Predictor . Our predictor leverages the graph neu- ral network family, wh...

    2013

  37. [45]

    |u| =|zt| = 8

    The dimensionality of latent is 8, i.e. |u| =|zt| = 8 . Location feature is represented as the center of entities|b| = 2, appearance feature|a| = 32. The region of each entity is set to a large enough fixed width and height to cover the entity,d = 70 in all of our experiment. A...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.