RECORD
Consistent Generative Query Networks
Computer Vision and Pattern Recognition · Submitted 5 Jul 2018
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Synthetic video evaluations confirm substantial gains in speed and functionality without loss in fidelity.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Stochastic video prediction models take in a sequence of image frames, and generate a sequence of consecutive future image frames. These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive. We introduce a model that overcomes these drawbacks by generating a latent representation from an arbitrary set of frames that can then be used to simultaneously and efficiently sample temporally consistent frames at arbitrary time-points. For example, our model can "jump" and directly sample frames at the end of the video, without sampling intermediate frames. Synthetic video evaluations confirm substantial gains in speed and functionality without loss in fidelity. We also apply our framework to a 3D scene reconstruction dataset. Here, our model is conditioned on camera location and can sample consistent sets of images for what an occluded region of a 3D scene might look like, even if there are multiple possibilities for what that region might contain. Reconstructions and videos are available at https://bit.ly/2O4Pc4R.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Stochastic video prediction models take in a sequence of image frames, and generate a sequence of consecutive future image frames. | background 1.00 | 0.07 | 0.03 | asserted 1.00 |
| s1 | These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive. | background 0.88 | 0.06 | 0.03 | asserted 0.98 |
| s2 | We introduce a model that overcomes these drawbacks by generating a latent representation from an arbitrary set of frames that can then be used to simultaneously and efficiently sample temporally consistent frames at arbitrary time-points. | contribution 0.89 | 0.92 | 0.52 | asserted 1.00 |
| s3 | For example, our model can "jump" and directly sample frames at the end of the video, without sampling intermediate frames. | method 0.42 | 0.86 | 0.08 | asserted 0.77 |
| s4 | Synthetic video evaluations confirm substantial gains in speed and functionality without loss in fidelity. | result 1.00 | 0.94 | 0.50 | asserted 1.00 |
| s5 | We also apply our framework to a 3D scene reconstruction dataset. | contribution 0.79 | 0.94 | 0.03 | asserted 0.97 |
| s6 | Here, our model is conditioned on camera location and can sample consistent sets of images for what an occluded region of a 3D scene might look like, even if there are multiple possibilities for what that region might contain. | contribution 0.62 | 0.93 | 0.06 | asserted 0.72 |
| s7 | Reconstructions and videos are available at https://bit.ly/2O4Pc4R. | contribution 0.43 | 0.38 | 0.02 | asserted 0.66 |
What was examined, and what was not
- arXiv listing abstract v3
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Stochastic video prediction models take in a sequence of image frames, and generate a sequence of consecutive future image frames. These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive. We introduce a model that overcomes these drawbacks by generating a latent representation from an arbitrary set of frames that can then be used to simultaneously and efficiently sample temporally consistent frames at arbitrary time-points. For example, our model can "jump" and directly sample frames at the end of the video, without sampling intermediate frames. Synthetic video evaluations confirm substantial gains in speed and functionality without loss in fidelity. We also apply our framework to a 3D scene reconstruction dataset. Here, our model is conditioned on camera location and can sample consistent sets of images for what an occluded region of a 3D scene might look like, even if there are multiple possibilities for what that region might contain. Reconstructions and videos are available at https://bit.ly/2O4Pc4R.
Human review
References
How the literature uses it
21 arXiv papers cite this in 33 sentences
Papers citing it · 12 sentences without a confident label
Uses its method or tool 3 papers
-
We then follow the CGQN paper [ref] and use ConvDraw [ref] to sample $$ from $r_$ and decode the target image $_q$ from $$ and query viewpoint $_q$.
ROOTS: Object-Centric Representation and Rendering of 3D Scenes · CGQN Baseline · 0.96
-
Another way of using the model is to extract a semantic embedding from the observations [ref] .
Distribution Guided Active Feature Acquisition · Ablations · 0.97
-
Another way of using the model is to extract a semantic embedding from the observations [ref] .
Active Feature Acquisition with Generative Surrogate Models · Ablations · 0.96
Compares with it 3 papers
-
In learning representations and rendering of 3D scenes, GQN [ref] and its variants [ref] are the most close to our work.
ROOTS: Object-Centric Representation and Rendering of 3D Scenes · Introduction · 0.80
-
We also demonstrate meta-transfer learning and improved generation quality in contrast to Consistent Generative Query Networks [ref] gained from the decoupling of temporal dynamics from the scene representations.
Sequential Neural Processes · Introduction · 0.99
-
[ref] and [ref] propose variants that are temporally consistent, but unlike us, they do not focus on the problem of modeling the forward dynamics.
S2RMs: Spatially Structured Recurrent Modules · Related Work · 0.91
Cites it as background 12 papers
-
This paper focuses on a family of neural rendering methods [ref] that infer the underlying 3D scene structure and faithfully produces the target view even at a distant query pose.
Sequential View Synthesis with Transformer · Introduction · 0.91
-
Generative Query Networks (GQN) (GQN) [ref] a framework to learn low dimensional embedding vectors that represent both the 3D scenes structure and appearance.
Continuous Object Representation Networks: Novel View Synthesis without Target View Supervision · Related Work · 0.89
-
Learning representations of 3D scenes or environments with partially observable inputs has been addressed by supervised learning [ref] , latent variable models [ref] , and generative adversarial networks [ref] .
Neural Multisensory Scene Inference · Related Works · 0.99
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.