REVIEW 4 major objections 5 minor 25 references
Inception-inspired LSTM for Next-frame Video Prediction
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read An Inception-inspired LSTM, replacing each convolutional LSTM gate with parallel 1x1, 3x3, and 5x5 convolutions, predicts next video frames more accurately than a standard convolutional LSTM.
desk verdict A plausible incremental architecture with a load-bearing parameter-count confound; the gains are real but not yet attributable to the multi-kernel design. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key object is the Inception-inspired LSTM cell: for each gate (input $i_t$, forget $f_t$, output $o_t$) and for the candidate update $g_t$, the pair $[x_t,h_{t-1}]$ is convolved in parallel by kernels of sizes $1\times1$, $3\times3$, and $5\times5$, and the three output stacks are concatenated and passed through the gate's activation. In version 2 the $5\times5$ convolution is replaced by two sequential $3\times3$ convolutions, matching the Inception-v2 factoring idea. The rest of the LSTM update ($c_t = f_t \odot c_{t-1} + i_t \odot g_t$, $h_t = o_t \odot \tanh(c_t)$) is unchanged, so the mechanism is a drop-in change to the gate internals that widens each gate's receptive-field range.
What would settle it
Train a convolutional LSTM matched in per-layer parameter count to the Inception LSTM's 6,595, for instance by widening its feature channels in the reported 3-48-96-192 configuration, and compare MSE and SSIM on the same KITTI and KTH splits under the same predictive-coding protocol. If the wider ConvLSTM matches or exceeds the Inception LSTM's scores, the attribution of the gain to multi-scale kernels is not supported.
Extended reading notes
Core claim
The central empirical claim is that an Inception-style replacement for the convolutional LSTM's gates yields better next-frame predictions. With four layers and the same predictive-coding protocol, Inception version 1 reports KITTI MSE of 0.0436 and SSIM of 0.868, against 0.0458 and 0.858 for the convolutional LSTM; on KTH the best version-1 MSE is 0.0103 with SSIM 0.964. Version 2 stays close to version 1 in accuracy while using fewer parameters because its $5\times5$ path is factored into two $3\times3$ convolutions. The paper interprets these results as evidence that a richer set of receptive-field scales, not a deeper network, improves spatiotemporal prediction.
Load-bearing premise
The paper attributes the performance gain to the multi-kernel design, even though the Inception LSTM has roughly six times more weights per layer than the ConvLSTM it is compared against and no equal-capacity ConvLSTM control is tested.
Editorial extensions
If this is right
- Kernel size no longer needs to be selected as a single hyperparameter for the recurrent gates; the Inception LSTM learns to combine $1\times1$, $3\times3$, and $5\times5$ views of the same input.
- Under the tested protocol, the Inception LSTM gives the best MSE and SSIM on both datasets among the three compared models, and version 1 is the overall best predictor.
- Version 2 offers a parameter and compute reduction of $7n_c$ per gate per layer relative to version 1 while retaining most of version 1's accuracy.
- Prediction quality in these models appears to plateau after about five frames of history, so the gate redesign does not shift the required memory horizon.
- Because the surrounding predictive-coding architecture and error-propagation rules are unchanged, the Inception gate can replace a convolutional LSTM without re-engineering the rest of the model.
Reading between the lines
- Because the Inception LSTM's per-layer parameter count (6,595) is roughly six times the ConvLSTM's (1,081) and no wider ConvLSTM baseline was run, the cleanest way to see whether the multi-scale kernels themselves matter is a parameter-matched comparison; the reported gap could partly be a capacity effect.
- A natural extension the paper does not pursue is to apply the same gate-level multi-scale idea to the recurrent weight path $h_{t-1}$ alone, or to a GRU cell, to see whether the benefit transfers beyond the convolutional-LSTM setting studied here.
- The version-1 versus version-2 comparison suggests that the effective receptive-field size contributes more than raw parameter count on these datasets; systematically varying the kernel set (e.g., $3\times3$, $5\times5$, $7\times7$) would map that trade-off directly.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an Inception-inspired convolutional LSTM for next-frame video prediction inside the PredNet predictive-coding framework. Two variants are defined: Inception v1 uses parallel 1x1, 3x3, and 5x5 convolutions in each LSTM gate, while Inception v2 replaces the 5x5 convolution with two stacked 3x3 convolutions. The method is compared with the standard convolutional LSTM on the KITTI and KTH data sets at two, three, and four layers, using MAE, MSE, and SSIM, with additional plots of metrics as a function of the number of previous frames. The authors report that the Inception-based LSTM outperforms the convolutional LSTM, that Inception v1 outperforms v2, and that v2 has lower computational cost. The source code is publicly released.
Significance. Taken at face value, the result would provide evidence that multi-scale convolutional kernels inside LSTM gates improve next-frame prediction, which is a modest architectural variation on PredNet with potential practical value. Strengths of the paper include the public code release, the use of standard datasets and metrics, and evaluation on held-out test data; there is no circularity in the sense of fitting the model on the test set. However, the central comparison is not yet established: the capacity difference between the Inception LSTM and the convolutional LSTM is unaddressed, the only confidence-interval evidence explicitly shows overlap, and the reported tables contain internal inconsistencies. The contribution is potentially useful but needs substantially stronger empirical support.
major comments (4)
- [Section 3, parameter-count paragraph] The central architectural claim is confounded by a large difference in model capacity: the parameter-count paragraph reports 6,595 parameters per layer for Inception LSTM version 1 versus 1,081 parameters for the convolutional LSTM, roughly a sixfold increase. The experiments do not include a wider ConvLSTM baseline with a matched parameter count, so the reported improvements over ConvLSTM cannot be attributed to the multi-kernel design as opposed to additional capacity. Please add a capacity-matched baseline, for example a wider ConvLSTM with comparable parameter count, before claiming that the Inception module is responsible for the gain.
- [Section 4, Figure 6] The only quantified uncertainty in the paper is the 95% confidence interval plot, and the accompanying text states that although the MSE of the Inception LSTM is lower than that of the convolutional LSTM, the confidence intervals overlap. This directly weakens the abstract's claim of outperformance. Moreover, the text identifies this figure as KITTI while the caption labels it KTH, so the reader cannot tell which dataset the statement refers to. Table 1 provides no error bars or significance tests; please report them and reconcile the figure label with the text.
- [Table 1] Table 1 contradicts the abstract and contains a likely data error. On KTH, Inception v2 (4L) has MSE 0.000463, which is lower than Inception v1 (4L) at 0.000524, contradicting the claim that Inception LSTM has better prediction performance than Inception v2 LSTM and the statement that Inception Version I shows the best performance. In addition, the KTH ConvLSTM (2L) row (MAE 0.044115, MSE 0.007191, SSIM 0.867645) is identical to the KITTI Inception v2 (4L) row, which suggests a copying or bookkeeping error and undermines confidence in the reported numbers as a whole. Please correct the table and rerun or verify the affected experiments.
- [Section 4, experimental setup] The paper does not specify the number of training and test sequences used for the KITTI and KTH evaluations, the data splits, the number of runs, or the random seeds. Many of the metric differences in Table 1 are in the fourth decimal place, so run-to-run variance could alter several of the conclusions. Please provide the missing protocol details and repeat measurements so that the ranking of models can be assessed.
minor comments (5)
- [Abstract] The abstract uses "unsupervised" while the full-text abstract says "self-supervised"; please use one term consistently.
- [Section 3, Eq. (3a)] The notation in Eq. (3a) is inconsistent with the stated design: a two-3x3 replacement of the 5x5 kernel should apply the second 3x3 convolution to the output of the first, but the equation introduces three 3x3 weight tensors (W1, W2, W3); please correct the equation.
- [Section 4] The claim that using Inception removes the need to choose a kernel size is overstated, since the method still fixes the kernel set {1x1, 3x3, 5x5}.
- [Related work, reference [9]] Reference [9] is listed as a "git book" with no author or title; it should be replaced by a citable source or removed.
- [Section 3, parameter-count derivation] The parameter count (1+9+25)*nc appears to count weights for only one side of the convolution, ignoring that both the input and the recurrent hidden state contribute weight tensors; please show the full count explicitly.
Circularity Check
No significant circularity: the reported comparisons are empirical evaluations on held-out test data with no fitted parameter renamed as a prediction.
full rationale
This paper is an empirical architecture comparison, not a derivation. The central claims (Inception LSTM outperforms ConvLSTM, and Inception v1 outperforms Inception v2) are supported by Table 1 and Figures 4 and 6, which report MAE, MSE, and SSIM on held-out KITTI and KTH test videos. There is no fitted parameter that is then called a prediction: the models are trained with a standard loss and evaluated with standard metrics on unseen frames. The mild observation that MSE is both the training cost function and one of the evaluation metrics is standard practice and does not make the evaluation circular. The paper cites prior work by the same authors ([6], [21]) only for unrelated topics such as video streaming and solar irradiance forecasting; these citations are not load-bearing for the Inception LSTM architecture or its evaluation. The concerns raised in the skeptic's analysis, such as the much larger parameter count of Inception LSTM relative to ConvLSTM and overlapping confidence intervals, are threats to the internal validity and statistical strength of the empirical claim, but they are not circularity: they do not show that the reported result is equivalent to its inputs by construction. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption PredNet predictive coding framework is a valid and representative baseline for next-frame video prediction.
- domain assumption Multi-kernel convolutions with sizes 1, 3, and 5 capture different motion magnitudes.
- domain assumption Hard sigmoid non-linearity and layer depths are appropriate for all compared models.
Cite this review
Pith. "Pith review of Inception-inspired LSTM for Next-frame Video Prediction." pith.science (2026). https://pith.science/paper/SVHRDR5T
@misc{pith2026190905622,
author = {Pith},
title = {Pith review of: Inception-inspired LSTM for Next-frame Video Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/SVHRDR5T}},
note = {Machine review of arXiv:1909.05622}
}
read the original abstract
The problem of video frame prediction has received much interest due to its relevance to many computer vision applications such as autonomous vehicles or robotics. Supervised methods for video frame prediction rely on labeled data, which may not always be available. In this paper, we provide a novel unsupervised deep-learning method called Inception-based LSTM for video frame prediction. The general idea of inception networks is to implement wider networks instead of deeper networks. This network design was shown to improve the performance of image classification. The proposed method is evaluated on both Inception-v1 and Inception-v2 structures. The proposed Inception LSTM methods are compared with convolutional LSTM when applied using PredNet predictive coding framework for both the KITTI and KTH data sets. We observed that the Inception based LSTM outperforms the convolutional LSTM. Also, Inception LSTM has better prediction performance compared to Inception v2 LSTM. However, Inception v2 LSTM has a lower computational cost compared to Inception LSTM.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Unsu- pervised learning for physical interaction through video prediction
FINN , C., G OODFELLOW , I., AND LEVINE , S. Unsu- pervised learning for physical interaction through video prediction. In Advances in Neural Information Process- ing Systems (2016), pp. 64–72
work page 2016
-
[2]
Deep visual foresight for planning robot motion
FINN , C., AND LEVINE , S. Deep visual foresight for planning robot motion. In Robotics and Automation (ICRA), 2017 IEEE International Conference on(2017), IEEE, pp. 2786–2793
work page 2017
-
[3]
A theory of cortical responses
FRISTON , K. A theory of cortical responses. Phil. Trans. R. Soc. B 360 (2005), 815–836
work page 2005
-
[4]
Vision meets robotics: The kitti dataset
GEIGER , A., L ENZ , P., S TILLER , C., AND URTASUN , R. Vision meets robotics: The kitti dataset. The In- ternational Journal of Robotics Research 32, 11 (2013), 1231–1237
work page 2013
-
[5]
K., K OUTN ´IK, J., S TE- UNEBRINK , B
GREFF , K., S RIVASTAVA, R. K., K OUTN ´IK, J., S TE- UNEBRINK , B. R., AND SCHMIDHUBER , J. Lstm: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems 28 , 10 (2017), 2222– 2232
work page 2017
-
[6]
HOSSEINI , M., S ALEHI , M. A., AND GOT- TUMUKKALA , R. Enabling interactive video streaming for public safety monitoring through batch scheduling. In 2017 IEEE 19th International Conference on High Performance Computing and Communications; IEEE 15th International Conference on Smart City; IEEE 3rd Figure 6: KTH data set next-frame prediction performance ...
work page 2017
-
[7]
An empirical exploration of recurrent network ar- chitectures
JOZEFOWICZ , R., Z AREMBA , W., AND SUTSKEVER , I. An empirical exploration of recurrent network ar- chitectures. In International Conference on Machine Learning (2015), pp. 2342–2350
work page 2015
-
[8]
KRIZHEVSKY , A., S UTSKEVER , I., AND HINTON , G. E. Imagenet classification with deep convolutional neural networks. In Advances in neural information pro- cessing systems (2012), pp. 1097–1105
work page 2012
Show all 25 references
-
[9]
git book, 2019
LEONARDOARAUJOSANTOS . git book, 2019
2019
-
[10]
Unsu- pervised learning of visual structure using predictive generative networks
LOTTER , W., K REIMAN , G., AND COX, D. Unsu- pervised learning of visual structure using predictive generative networks. arXiv preprint arXiv:1511.06380 (2015)
2015 arXiv
-
[11]
Deep pre- dictive coding networks for video prediction and un- supervised learning
LOTTER , W., K REIMAN , G., AND COX, D. Deep pre- dictive coding networks for video prediction and un- supervised learning. arXiv preprint arXiv:1605.08104 (2016)
2016 arXiv
-
[12]
R., AND SAVAKIS, A
MEDEL , J. R., AND SAVAKIS, A. Anomaly detection in video using predictive convolutional long short-term memory networks. arXiv preprint arXiv:1612.00390 (2016)
2016 arXiv
-
[13]
Spatio-temporal video autoencoder with differentiable memory
PATRAUCEAN , V., H ANDA , A., AND CIPOLLA , R. Spatio-temporal video autoencoder with differentiable memory. arXiv preprint arXiv:1511.06309 (2015)
2015 arXiv
-
[14]
P., AND BALLARD , D
RAO, R. P., AND BALLARD , D. H. Predictive cod- ing in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neu- roscience 2, 1 (1999), 79
1999
-
[15]
Recog- nizing human actions: a local SVM approach
SCH ¨ULDT , C., L APTEV , I., AND CAPUTO , B. Recog- nizing human actions: a local SVM approach. In Proc. Int. Conf. Pattern Recognition (ICPR’04) (Cambridge, U.K, 2004)
2004
-
[16]
A comprehensive survey on deep future frame video prediction
SELVA CASTELL ´O, J. A comprehensive survey on deep future frame video prediction. Master’s thesis, Univer- sitat Polit`ecnica de Catalunya, 2018
2018
-
[17]
Very deep con- volutional networks for large-scale image recognition
SIMONYAN , K., AND ZISSERMAN , A. Very deep con- volutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
2014 arXiv
-
[18]
Going deeper with convolutions
SZEGEDY , C., L IU, W., J IA, Y., S ERMANET , P., REED , S., A NGUELOV , D., E RHAN , D., V AN- HOUCKE , V., AND RABINOVICH , A. Going deeper with convolutions. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (2015), pp. 1–9
2015
-
[19]
Rethinking the inception architec- ture for computer vision
SZEGEDY , C., V ANHOUCKE , V., I OFFE , S., S HLENS , J., AND WOJNA , Z. Rethinking the inception architec- ture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (2016), pp. 2818–2826
2016
-
[20]
Decomposing motion and content for natural video sequence prediction
VILLEGAS , R., Y ANG , J., H ONG , S., L IN, X., AND LEE, H. Decomposing motion and content for natural video sequence prediction. arXiv preprint arXiv:1706.08033 (2017)
2017 arXiv
-
[21]
WOJTKIEWICZ , J., H OSSEINI , M., G OTTUMUKKALA , R., AND CHAMBERS , T. L. Hour-ahead solar irra- diance forecasting using multivariate gated recurrent units. Energies 12, 21 (2019), 4055
2019
-
[22]
Convolutional LSTM network: A machine learning approach for pre- cipitation nowcasting
XINGJIAN , S., C HEN , Z., W ANG , H., Y EUNG , D.- Y., W ONG , W.-K., AND WOO, W.- C. Convolutional LSTM network: A machine learning approach for pre- cipitation nowcasting. In Advances in neural informa- tion processing systems (2015), pp. 802–810
2015
-
[23]
End-to- end learning of driving models from large-scale video datasets
XU, H., G AO, Y., Y U, F., AND DARRELL , T. End-to- end learning of driving models from large-scale video datasets. In 2017 IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR) (2017), IEEE, pp. 3530–3538
2017
-
[24]
Wide resid- ual networks
ZAGORUYKO , S., AND KOMODAKIS , N. Wide resid- ual networks. arXiv preprint arXiv:1605.07146 (2016)
2016 arXiv
-
[25]
ZHOU , Y., AND BERG , T. L. Learning temporal trans- formations from time-lapse videos. In European Con- ference on Computer Vision (2016), Springer, pp. 262– 277
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.