{"id":"b8c72622-0b76-4d83-b25e-3ea1b17f6915","arxiv_id":"2411.10198","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A fully convolutional spatio-temporal patch mixer, STLight, matches or beats RNN-based baselines on predictive learning benchmarks at a fraction of the parameter and FLOP budget.","lead":"STLight is a fully convolutional architecture for video frame prediction that interleaves past frames into spatio-temporal patches and processes them with a compact mixer network. It reports lower error and fewer parameters and FLOPs than many recurrent and non-recurrent baselines on Moving MNIST, TaxiBJ, and KTH benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA accuracy claim is contradicted by the paper's own Table 6: at 2000 epochs, PredRNN++ beats STLight-L on MMNIST MSE and SSIM, so the unqualified SOTA claim is not robust.","rationale":"The reader identified the faithfulness of OpenSTL baselines as the weakest assumption. I agree that this is a serious concern, but the paper itself provides stronger, internal evidence that the SOTA accuracy claim is not robust: in Appendix A Table 6, under 2000-epoch training, a baseline (PredRNN++) achieves better MSE and SSIM than STLight-L on MMNIST. This is not a matter of external baseline quality; it is a direct contradiction within the manuscript's own results. The efficiency advantage (FLOPs, parameters) is credible and well-supported, so the paper remains a useful architectural contribution. However, the unqualified claim of 'state-of-the-art performance across different datasets and settings' is too strong. The verdict should remain CONDITIONAL: the authors should qualify the SOTA claim to the standard OpenSTL protocol and provide error bars, or the abstract should be revised to an accuracy-efficiency tradeoff claim. I did not find an internal inconsistency in the architecture or complexity analysis that would invalidate the method itself; the concern is specifically about the empirical claim's robustness and framing.","tokens_in":18543,"tokens_out":9501,"duration_ms":83742,"concrete_test":"Reproduce the 2000-epoch MMNIST comparison from Table 6: run PredRNN++, TAU, MIM, and STLight-L under identical OpenSTL training schedules with at least 3 random seeds, using the same hardware and the reported hyperparameters. If PredRNN++ consistently yields lower MSE and higher SSIM than STLight-L, the abstract's SOTA claim should be revised to a conditional 'SOTA under standard 200-epoch OpenSTL settings' or to a tradeoff claim; additionally, report error bars for all Table 3 comparisons.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim, stated in the abstract and contributions, is that STLight achieves state-of-the-art accuracy while improving efficiency. Table 3 supports this under the standard OpenSTL training budget (e.g., 200 epochs on MMNIST). However, the paper's own extended-training evaluation (Appendix A, Table 6) reports that after 2000 epochs on MMNIST, PredRNN++ attains MSE 14.07 and SSIM 0.9698, both better than STLight-L's MSE 14.77 and SSIM 0.9686. STLight-L still has a large FLOP advantage (32.9G vs 171.7G), so the efficiency benefit is real, but the 'outperforms or matches previous methods in accuracy' contribution is not supported in this setting. Appendix A also concedes that OpenSTL does not guarantee correctness, protocol adherence, or optimality of baseline training parameters. Since the SOTA conclusion in Table 3 rests entirely on those baseline numbers, and the paper's own data show the accuracy ordering can invert with training duration, the unqualified SOTA claim is fragile. At minimum, the claim must be qualified to the standard OpenSTL protocol and framed as an accuracy-efficiency tradeoff, not as unconditional SOTA accuracy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes STLight, a fully convolutional architecture for spatio-temporal predictive learning (STL). It encodes input frame sequences into overlapping spatio-temporal patches using a single convolution, processes these patches with repeated STLMixer blocks that combine local and dilated depthwise convolutions, and decodes via PixelShuffle and a 1x1 convolution. The authors report accuracy and efficiency results on Moving MNIST, TaxiBJ, KTH, and a cross-dataset generalization test (KITTI-to-Caltech), claiming state-of-the-art accuracy with substantially lower parameter counts and FLOPs than recurrent baselines. The paper includes ablations of the encoder, decoder, mixer, skip connections, and hyperparameters, plus an appendix with extended training, an alternative literature comparison, and the full PyTorch implementation.","tokens_in":18848,"tokens_out":4746,"duration_ms":43912,"significance":"If the accuracy claims were fully supported, STLight would be a notable contribution: it demonstrates that a simple, fully convolutional design can be competitive with recurrent models in predictive learning while being markedly more efficient. The efficiency gains are real and well documented in the parameter/FLOP tables, and the public code and appendix implementation strengthen reproducibility. The ablation study and sample-efficiency analysis are useful. However, the central 'state-of-the-art accuracy' claim is not consistently supported by the paper's own results, and the contribution should be reframed as an efficient architecture that is competitive under specific benchmark protocols rather than unconditionally SOTA.","major_comments":[{"comment":"The abstract and contribution list state that STLight 'achieves state-of-the-art performance' and 'outperforming or matching previous methods in terms of accuracy'. This is contradicted by the paper's own extended-training results in Appendix A, Table 6: at 2000 epochs on MMNIST, PredRNN++ attains MSE 14.07 and SSIM 0.9698, both better than STLight-L's MSE 14.77 and SSIM 0.9686. The claim must be qualified to the specific training settings (standard OpenSTL protocol, 200 epochs) and should be framed as an accuracy-efficiency tradeoff, not unconditional SOTA accuracy.","section":"Abstract and §4.3"},{"comment":"The text in §4.2.1 says STLight 'outperforms both recurrent-based and recurrent-free models across all metrics'. This is not accurate: on TaxiBJ, TAU achieves lower MAE (14.93 vs STLight-L's 15.00), and on KTH, PredRNN++ achieves higher PSNR (28.13 vs 27.57) and SSIM (0.9124 vs 0.9113). The wording should be corrected to reflect that STLight is best or near-best on some metrics and datasets but not all.","section":"§4.2.2 and Table 3"},{"comment":"The narrative in Appendix A misreports its own data. The text claims that 'our model continues to outperform recurrent architectures like PredRNN++ and MIM in accuracy', but Table 6 shows PredRNN++ with lower MSE (14.07 vs 14.77) and higher SSIM (0.9698 vs 0.9686) than STLight-L. The appendix is meant to corroborate the main-table conclusions, yet it actually demonstrates that the accuracy ordering can invert with training duration. This internal inconsistency is load-bearing for the paper's central claim and must be fixed.","section":"Appendix A, Table 6 and surrounding text"},{"comment":"The main results in Tables 3, 5, and 6 are single runs with no error bars, while the baselines are taken from the OpenSTL benchmark. The authors themselves acknowledge in Appendix A that OpenSTL 'does not fully guarantee (1) the correctness of the implementations, (2) the adherence to the original training protocols of each baseline, or (3) the optimality of the default standard training parameters'. Given this caveat and the small differences on some metrics (e.g., KTH SSIM 0.9113 vs 0.9124), the term 'state-of-the-art' should be replaced by a more circumspect formulation such as 'competitive under the OpenSTL evaluation protocol', and error bars or multiple-seed results should be provided for the main comparisons.","section":"§4.1 Train-eval settings and Appendix A"}],"minor_comments":[{"comment":"The last row lists 'STLight-L (Ours) 32.9M 32.9M'; the second occurrence is clearly a typo and should be 32.9G for FLOPs.","section":"Table 6"},{"comment":"Equation (1) uses an unclear notation for indexing frames; it should be written explicitly as X ∈ R^{T×C×H×W} with a definition that the frames range from t0−T+1 to t0.","section":"Equation (1)"},{"comment":"In §4.5, the text refers to 'Figure 7' for the kernel-size study, but the actual figure is in Appendix C; please make the cross-reference explicit.","section":"§4.5 and Appendix C"},{"comment":"The text says 'In Tables 8 and ??' — the placeholder '??' should be replaced with the actual table number.","section":"Appendix E"},{"comment":"The phrase 'state-of-the-art' appears in the abstract and contributions without specifying the benchmark or training protocol; please qualify it consistently (e.g., 'under the OpenSTL training settings') to match the actual evidence.","section":"Throughout"},{"comment":"The claim 'For the first time in the STL context, we jointly process temporal and spatial dynamics' may be overstated given earlier joint spatio-temporal processing in 3D convolution and video diffusion models; consider softening the novelty claim or citing comparative prior work.","section":"§1 Contributions"}],"recommendation":"major_revision","confidential_remarks":"The core issue is that the paper's headline accuracy claim is contradicted by its own extended-training experiments (Table 6), which is likely to be noticed by any careful reader. The authors should be asked to rework the abstract, contributions, and relevant discussion to present STLight as an efficient fully-convolutional model that is competitive with (not uniformly better than) prior methods under the standard OpenSTL protocol. The efficiency and architectural contributions are solid enough to warrant a revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"STLight is a real contribution to the efficient-video-prediction line of work, and the code and ablations make it worth engaging with. The abstract's SOTA claim, though, is broader than the evidence supports.\n\nWhat's genuinely new: instead of encoding frames independently, STLight stacks the input along the channel axis and uses one convolution to make overlapping spatio-temporal patches. The STLMixer is a lightweight ConvMixer variant with local plus dilated depthwise convolutions, and decoding uses PixelShuffle plus a 1x1 conv. Interleaving frames in channels was used by Voleti et al. for video diffusion, but this specific combination for STL is new, and the paper evaluates it carefully. The efficiency numbers are the strongest part: STLight-L gets MMNIST MSE 21.80 with 32.9G FLOPs versus PredRNN++'s 22.06 at 171.7G under the standard OpenSTL budget. The extensive ablations on patch size, overlap, kernel sizes, initialization, and decoding order are useful, and the public OpenSTL configs make reproduction straightforward. The complexity analysis in Appendix B is also clean.\n\nSoft spots: (1) The SOTA accuracy claim is not robust. On KTH, STLight-L leads MAE but trails PredRNN++ on PSNR and SSIM. In the paper's own 2000-epoch MMNIST table, PredRNN++ reaches MSE 14.07 and SSIM 0.9698, both better than STLight-L's 14.77 and 0.9686. The Appendix text saying STLight 'continues to outperform' PredRNN++ is misreading its own table. The defensible claim is competitive accuracy at much lower cost under standard training budgets. (2) All baseline comparisons rest on OpenSTL's public numbers; Appendix A honestly notes OpenSTL may not match original implementations. That is acceptable for a benchmark paper but should be part of the main-text claim, not buried. (3) Main results are single runs without error bars, and hyperparameters are tuned by grid search on the same benchmarks. This is common in the field but makes the margins over baselines look firmer than they are. Minor: Table 6 lists STLight-L FLOPs as 32.9M; it should be 32.9G.\n\nBottom line: this is a solid empirical architecture paper, not a breakthrough. It deserves a serious referee and, after the accuracy claims are qualified and the typo fixed, likely acceptance. I'd bring it to reading group; the OpenSTL-baseline-fidelity discussion alone is worth having.","headline":"Solid efficient fully-conv video predictor with genuine gains; accuracy claims need reining in before publication.","tokens_in":19348,"tokens_out":3758,"would_cite":true,"duration_ms":36936,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"STLight claims a purely convolutional network can match or beat recurrent and attention-based video predictors on standard benchmarks while using far fewer parameters and FLOPs.","keywords":["spatio-temporal predictive learning","video prediction","fully convolutional network","depthwise convolution","spatio-temporal patches","pixel shuffle","self-supervised learning","efficient video prediction"],"falsifier":"Re-train each baseline with its original authors' training protocols and optimal hyperparameters rather than the benchmark defaults; if tuned ConvLSTM, PredRNN, or TAU then beat STLight's reported MSE and SSIM at comparable FLOPs, the claimed state-of-the-art status fails.","tokens_in":18388,"feed_emoji":"🎞️","tokens_out":6068,"duration_ms":54970,"temperature":0.7,"pith_summary":"This paper tries to establish that a purely convolutional network, with no recurrence and no attention, can match or beat the best recurrent and recurrent-free models on standard spatio-temporal prediction benchmarks while using far fewer parameters and FLOPs. The route is to stop encoding frames independently. Instead, STLight stacks all observed frames along the channel dimension and carves them into overlapping spatio-temporal patches with a single convolution, so each patch token carries both where things are and when they happen. A mixer of depthwise convolutions then combines local and distant patch relationships before a parameter-free pixel shuffle rebuilds the predicted frames. If the claim holds, it removes the main argument for recurrent units in video prediction and gives resource-constrained settings a simpler, scalable baseline.","feed_headline":"All-convolutional model beats recurrent video predictors at low cost","feed_subtitle":"STLight stacks past frames into spatio-temporal patches, then mixes near and far details with no recurrence or attention.","key_machinery":"The spatio-temporal patch: frames stacked in the channel dimension are convolved with overlapping patches, so each output token encodes spatial position and temporal phase together, and a single convolution replaces the separate spatial encoder and temporal translator of standard Spatial-Temporal-Spatial designs. The STLMixer block is a ConvMixer-style unit with two depthwise convolutions, a small kernel for local detail and a dilated larger kernel for distant context, followed by a 1x1 pointwise convolution; repeated blocks with inter-block skip connections let near and far patches exchange information. The decoder's PixelShuffle operation is parameter-free and rearranges patch dimensions back to full resolution before a single 1x1 convolution reassembles the output frames.","core_discovery":"The central claim is that the bottleneck in convolutional video prediction is not convolution itself, but the habit of treating frames as separate spatial inputs. STLight interleaves the T observed frames along the channel dimension into a single tensor and applies one convolution to form overlapping patches whose hidden dimension mixes spatial and temporal content. The resulting STLMixer stack alternates a small-kernel depthwise convolution and a dilated larger-kernel depthwise convolution, with a pointwise mixer, to capture both close and distant patch interactions without attention or recurrence. The decoder is a PixelShuffle followed by one 1x1 convolution, so the only learnable layers in the whole model are channel-wise and depth-wise convolutions. On Moving MNIST, TaxiBJ and KTH, the authors report that STLight variants match or beat published state-of-the-art results while using a fraction of the parameters and FLOPs of recurrent baselines, and they report better sample efficiency and cross-dataset generalisation on KITTI-to-Caltech.","pith_inferences":["If the public baseline numbers the paper compares against are re-tuned to their original authors' protocols, the size of STLight's advantage may shrink; the paper itself notes that its benchmark suite does not guarantee faithful, optimally tuned baselines.","The spatio-temporal patch recipe could transfer to other frame-sequence prediction tasks, such as radar nowcasting or surgical video, where channel-stacking frames is already a common preprocessing step.","The claim that wide dilated depthwise convolutions can substitute for attention in this setting suggests a cheaper inductive bias for long-horizon video prediction; a direct test would swap the STLMixer's dilated kernel for an attention block at equal parameter budget.","Because the decoder is a parameter-free PixelShuffle plus one 1x1 convolution, the architecture is unusually easy to port to edge and mobile accelerators, an implication the paper gestures at only through FLOP counts."],"forward_implications":["Recurrent units are not required for state-of-the-art frame prediction; channel-wise interleaving of frames plus a single convolution gives temporal characterisation.","Resource-constrained applications can deploy competitive predictors at a fraction of the FLOPs: for example, STLight-S on Moving MNIST uses about 14% of the FLOPs of PredRNN while achieving a lower MSE.","Long-sequence prediction, such as KTH 20-frame forecasting, can be done in one fully parallel forward pass rather than recursive rollouts, with STLight-L matching PredRNNv2 using roughly 2% of its FLOPs.","Cross-dataset generalisation from KITTI to Caltech improves with STLight scale, with strong results across the 0.1M to 15M parameter range.","Training is sample-efficient: STLight reaches lower MSE in fewer epochs than recurrent and recurrent-free baselines under identical schedules."],"supporting_citations":[{"why":"SimVP supplies the recurrent-free baseline framework and the encoder/decoder design that STLight ablates against.","marker":"[11]"},{"why":"PixelShuffle provides the parameter-free decoder operation that STLight uses to restore spatial resolution.","marker":"[28]"},{"why":"ConvLSTM is a recurrent baseline that STLight must beat on Moving MNIST, TaxiBJ, and KTH.","marker":"[29]"},{"why":"TAU is the attention-based recurrent-free state of the art that STLight compares against and replaces in ablations.","marker":"[33]"},{"why":"The benchmark suite supplies the uniform training settings and published baseline numbers that all comparisons rely on.","marker":"[34]"},{"why":"ConvMixer is the mixer architecture that the STLMixer block extends with a two-stage depthwise convolution.","marker":"[37]"},{"why":"PredRNN is a recurrent baseline whose parameters and FLOPs STLight uses to demonstrate efficiency gains.","marker":"[43]"}],"fun_headline_variants":["Convolutions only: STLight predicts video frames faster and cheaper","STLight: mixing space-time with one conv, no recurrence, less compute","Video prediction without RNNs: STLight does it all with convolutions","STLight flattens time into channels, cutting FLOPs and parameters","All-convolutional video predictor: STLight outperforms RNNs at lower cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison rests on the assumption that the public benchmark numbers used for every baseline are faithful, well-tuned implementations of the original methods; the paper itself notes in its appendix that the benchmark suite does not guarantee correctness of implementations, adherence to original training protocols, or optimality of default training parameters.","fun_headline_variants_meta":{"raw":{"variants":["Convolutions only: STLight predicts video frames faster and cheaper","STLight: mixing space-time with one conv, no recurrence, less compute","Video prediction without RNNs: STLight does it all with convolutions","STLight flattens time into channels, cutting FLOPs and parameters","All-convolutional video predictor: STLight outperforms RNNs at lower cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000403,"raw_usage":{"total_tokens":2114,"prompt_tokens":972,"completion_tokens":1142,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1041}},"tokens_in":588,"tokens_out":1142,"duration_ms":8447,"temperature":1.0,"reasoning_tokens":1041,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:51:04.919801+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train each baseline with its original authors' training protocols and optimal hyperparameters rather than the benchmark defaults; if tuned ConvLSTM, PredRNN, or TAU then beat STLight's reported MSE and SSIM at comparable FLOPs, the claimed state-of-the-art status fails.","supporting_citations":[{"cited_title":"Simvp: Simpler yet better video prediction","cited_arxiv_id":null,"evidence_quote":"SimVP supplies the recurrent-free baseline framework and the encoder/decoder design that STLight ablates against."},{"cited_title":"Convolutional lstm network: A machine learning approach for precipitation nowcasting","cited_arxiv_id":null,"evidence_quote":"ConvLSTM is a recurrent baseline that STLight must beat on Moving MNIST, TaxiBJ, and KTH."},{"cited_title":"Temporal attention unit: To- wards efficient spatiotemporal predictive learning","cited_arxiv_id":null,"evidence_quote":"TAU is the attention-based recurrent-free state of the art that STLight compares against and replaces in ablations."},{"cited_title":"Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms","cited_arxiv_id":null,"evidence_quote":"PredRNN is a recurrent baseline whose parameters and FLOPs STLight uses to demonstrate efficiency gains."}],"review_version":1}