{"id":"fa567149-f6c9-4d3e-be9a-10321e25ca8e","arxiv_id":"2505.02159","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A training strategy that reuses long-sequence hidden states during short-clip backpropagation, combined with ReLU-squared attention and a gated FFN, lifts video super-resolution accuracy to a reported state of the art.","lead":"This paper trains recurrent video super-resolution models on short clips while reusing hidden states precomputed from a full forward pass over long sequences, and it adds a transformer block that filters temporal features with ReLU-squared attention and a gated feed-forward unit. The method reports top PSNR on the REDS4 and ToS3 benchmarks with lower FLOPs than the closest prior model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-time numbers for the truncated-backprop strategy in Table 2 conflict with the per-iteration compute implied by Algorithm 1, undermining the central efficiency claim.","rationale":"The paper's central claim is that LRTI-VSR achieves state-of-the-art performance while maintaining training and computational efficiency. The most load-bearing premise is therefore not the theoretical quality of bootstrapped hidden states, which is empirically supported by Table 2 (TB improves from 31.08 to 31.44 dB when training from scratch with identical clip length), but the quantitative efficiency evidence. Table 2's GPU-day figures are internally inconsistent with Algorithm 1's per-iteration workload for batch size 1: the TB configuration processes at least 5-7.5× the frame-steps of the L=8 baseline and more than the L=24 configuration, yet it is reported nearly as fast as L=8 and faster than L=24. No implementation detail in the paper resolves this; caching the long-forward hidden states would violate the algorithm's requirement of current-parameter states, and the text explicitly sets batch size to 1. If the timings are wrong, the 'nearly 2.5× acceleration' and 'minor increase in training overhead' statements are unsupported, which matters for the paper's contribution as a training-efficiency method. The reader's identified weakness (bootstrapped hidden states) is real but empirically mitigated by the from-scratch ablations and the generic gains on BasicVSR/PSRT, so I do not believe it is the single most load-bearing issue. The training-time inconsistency is concrete, checkable, and directly tied to the headline efficiency claim, so the verdict should remain conditional pending a corrected timing report or a reproduction.","tokens_in":15913,"tokens_out":16180,"duration_ms":203841,"concrete_test":"Clone the released repo and instrument the training loop to count forward/backward frame-steps per iteration. Run each Table 2 configuration (L=8; TB T=40,L=8,N=5; L=24; L=40) for a fixed 5,000 iterations on one RTX4090 with batch size 1, measuring wall-clock time. If the measured time ratios do not match the frame-step ratios (TB should be slower than L=24 per iteration), then the GPU-day column is erroneous or the algorithm's workload is misstated; the claimed '2.5×' speedup over L=24 training should be revised accordingly.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Algorithm 1 states that each TB iteration runs a full forward pass over a long video (T frames) and then N short-clip forward/backward passes. The paper's own Table 2 protocol uses batch size 1 and 300K iterations for all rows. For the TB row (T=40, L=8, N=5), each iteration processes 40 forward steps plus 5 clips × (8 forward + 8 backward) steps ≈ 120 frame-steps, versus the L=8 baseline's 16 frame-steps and the L=24 baseline's 48 frame-steps (forward+backward). Yet the reported GPU days are 3.33 (L=8), 3.75 (TB L=8), and 9.12 (L=24). This implies TB is only 1.13× slower than L=8 and 2.4× faster than L=24, both impossible under the stated per-iteration workloads unless the long forward pass is nearly free or the baselines use far larger batches than the 'batch size 1' description. The same tension appears in Table 1 if the baseline is run at batch size 1. Because the abstract and §4.2 explicitly advertise 'training and computational efficiency' and a 'nearly 2.5×' acceleration, this internal inconsistency threatens a central contribution. The performance gains themselves may be reproducible, but the efficiency evidence requires correction or clarification.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LRTI-VSR, a training and architecture framework for recurrent video super-resolution. The main contributions are (i) a truncated-backpropagation-style training strategy that first runs a forward pass over a long video to obtain hidden states and then performs backpropagation on short clips conditioned on those hidden states, and (ii) a Refocused Intra&Inter-frame Transformer Block (RITB) that replaces softmax attention with a ReLU2 activation and injects the previous-frame aligned hidden state into the FFN through a Refocused Gated Unit (RGU). The method is evaluated on REDS4 and ToS3 for 4x VSR, reporting 33.06 dB PSNR on REDS4 and 34.81 dB on ToS3, and the training strategy is shown to improve BasicVSR, BasicVSR++, and PSRT. The paper also reports training time and memory comparisons, claiming a nearly 2.5x training acceleration relative to training on 24-frame clips.","tokens_in":16262,"tokens_out":8476,"duration_ms":90836,"significance":"If the efficiency claims are accurate, the proposed training strategy would be a broadly useful and generic technique for recurrent VSR, since it is validated on multiple external architectures (BasicVSR, BasicVSR++, PSRT) and the RITB block improves over the authors' own prior IIAB baseline. The paper releases code, uses standard benchmarks, and provides component ablations that are generally clean. However, the central efficiency claim is internally inconsistent with the algorithmic description, and the training strategy relies on an unanalyzed stale-hidden-state mechanism. These issues must be resolved before the contribution can be fully assessed.","major_comments":[{"comment":"The reported GPU days are inconsistent with the per-iteration workload implied by Algorithm 1. With T=40, L=8, N=5, each iteration requires a full forward pass over 40 frames (plus a backward-direction pass if the bidirectional model is used) and then five short-clip forward/backward passes over 8 frames, i.e., at least 120 frame-steps per iteration, versus 16 for the L=8 baseline and 48 for the L=24 baseline. The reported times (3.75 vs 3.33 and 9.12 GPU days for 300K iterations with batch size 1) imply that the TB training is only 13% slower than L=8 and 2.4x faster than L=24, which is arithmetically impossible under the stated protocol unless the long forward pass is amortized, performed once per epoch, or excluded from the timing. The abstract and Section 4.2 explicitly advertise 'nearly 2.5x acceleration' and 'training and computational efficiency,' so this discrepancy directly affects a central claim; please clarify the timing protocol or correct the efficiency numbers.","section":"Section 4.2 (Table 2) and Algorithm 1"},{"comment":"The long forward pass computes hidden states H with the current model parameters, after which N short-clip backward passes update the weights while H remains fixed. Therefore, after the first clip update, all subsequent clips in the same outer iteration are trained against stale hidden states from the pre-update model. The paper does not discuss or analyze this bootstrapping mismatch, nor does it provide a sensitivity study with respect to N. Because the entire efficiency argument rests on this strategy, please provide a justification (e.g., why staleness is benign for N up to 6 or 7) or an empirical analysis showing the effect of N on convergence and final accuracy.","section":"Algorithm 1 (lines 3-10)"},{"comment":"The ablations and hyperparameter choices are evaluated on REDS4, the same test set used for the final comparison in Table 3, and no error bars or multi-seed runs are reported. The SSIM gains in Table 1 are very large relative to the PSNR gains (e.g., BasicVSR: +0.14 dB PSNR but +0.0151 SSIM; BasicVSR++: +0.10 dB but +0.0107 SSIM), which is unusual and suggests either a systematic bias in the evaluation protocol or a typo. Please clarify the evaluation procedure and, ideally, validate the training strategy on a held-out set (e.g., ToS3) to guard against overfitting to REDS4.","section":"Section 4.2 and Table 1"}],"minor_comments":[{"comment":"The rows after 'RITB' are not labeled with method names; the checkmarks are placed ambiguously, making it difficult to read which configuration corresponds to each row. Please reformat the table with explicit labels (e.g., 'RITB w/o RGU', 'RITB w/o ReLU2') and align the checkmarks correctly.","section":"Table 4"},{"comment":"There are several typos and grammatical issues: 'paopagation' in Figure 1's caption, 'dateset' in Section 4.1, 'to to build' in Section 4.2, 'an reduction' in Section 4.2, and 'Transformaer-based' in the supplementary material. Please proofread the manuscript.","section":"Throughout"},{"comment":"The equation defines \\hat{I}^{HR}_{clip} as the output of the model, but the text describing it says 'the output of the forward propagation process during the VSR model's training.' Consider clarifying that the short-clip forward pass also uses the hidden states H_clip as inputs, as shown in the algorithm.","section":"Section 3.2, Eq. (3)"},{"comment":"The main text states that all models are trained from scratch for 600K iterations, but the supplementary material specifies a batch size of 24 for the final model. The main text should state the batch size as well, especially since Table 2 uses batch size 1 and the difference in batch size may affect the reported GPU times.","section":"Section 4.1 and Supplementary C.2"}],"recommendation":"major_revision","confidential_remarks":"The efficiency inconsistency in Table 2 is the main blocker; if the authors clarify that the long forward pass is amortized over multiple iterations or performed once per epoch (rather than once per iteration), the efficiency claim may become defensible. The stale-hidden-state issue is also important but could be addressed empirically. The paper's generic training strategy is validated on external architectures, which is a genuine strength, and the RITB improvements are cleanly ablated. I would be willing to look at a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on 2505.02159. The paper's real contribution is a training strategy: run a full recurrent forward pass over a long video to get hidden states, then backprop through short clips with those states as context. That's TBPTT applied to recurrent VSR, and the authors validate it on BasicVSR, BasicVSR++, and PSRT, getting 0.10-0.14 dB gains. That cross-architecture transfer is solid evidence the idea works. The RITB block is a smaller but clean improvement: ReLU2 in place of softmax plus a gated FFN that injects the previous hidden state. The ablations support each piece.\n\nThe soft spots. First, the efficiency numbers in Tables 1 and 2 don't square with Algorithm 1. If each iteration really does a forward pass over T=40 frames and then N=5 clip trainings of length 8, the per-iteration compute should be roughly 7-8x the L=8 baseline, not the 1.13x the GPU days suggest. The same pattern appears in Table 1. Either the long forward pass is amortized or cached and the paper doesn't say so, or the GPU days are misreported. Since the abstract and §4.2 advertise 'training and computational efficiency' and 'nearly 2.5x acceleration', this is load-bearing. The performance gains might be reproducible, but the efficiency claim needs a correction or a much clearer description of the actual loop.\n\nSecond, all ablations and hyperparameter choices are judged on REDS4, the same test set used for the final numbers, and there are no error bars or multi-seed runs. That's common in this subfield but it does mean the 0.1-0.2 dB differences between components could partly be noise.\n\nOverall, the training idea is worth knowing and the RITB block is a reasonable incremental improvement. The SOTA numbers are plausible, though I'd want the efficiency question resolved before relying on them. I'd send this to peer review, but with a demand to fix the timing analysis and report some measure of variance.\n\nBest.","headline":"A useful TBPTT-for-VSR training idea and a clean RITB block, but the reported training-time numbers don't match Algorithm 1; the efficiency claim needs a fix before I'd trust it.","tokens_in":16778,"tokens_out":7467,"would_cite":true,"duration_ms":81943,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Recurrent video super-resolution can learn long-range temporal detail by backpropagating through short clips while a single forward pass supplies full-video context.","keywords":["video super-resolution","recurrent networks","truncated backpropagation through time","long-range temporal information","transformer","refocused attention","ReLU2 activation","REDS dataset"],"falsifier":"Train the same recurrent VSR architecture on REDS4 under two protocols matched in model, clip length, sampling schedule, and iterations: the paper's one-forward-pass-per-outer-loop protocol, and a version that recomputes the long-video hidden states after every short-clip gradient update. If the recomputed-context version outperforms the paper's by a clear margin, the fixed-context assumption is the bottleneck; if the two are close, a single forward pass is sufficient context.","tokens_in":15728,"feed_emoji":"🎬","tokens_out":13283,"duration_ms":129386,"temperature":0.7,"pith_summary":"Recurrent video super-resolution models get sharper results when they see long-range temporal context, but training them on long clips is memory-hungry and slow. This paper argues that a model can learn those long-range patterns while training on short clips: run one forward pass over the whole long video to obtain hidden states, then sample short clips and backpropagate only through them, using the precomputed context from just outside each clip. The paper also introduces a transformer block whose attention uses $\\mathrm{ReLU}^2$ instead of SoftMax and whose feed-forward network is gated by the aligned previous-frame hidden state, so the model suppresses misaligned temporal features and concentrates on useful ones. The paper reports that the strategy is generic: it improves BasicVSR, BasicVSR++, and PSRT by 0.10–0.14 dB on REDS4, and the complete LRTI-VSR model reaches 33.06 dB on REDS4 and 34.81 dB on ToS3 at $4\\times$ super-resolution.","feed_headline":"Short clips plus full-video context lift VSR to 33.06 dB","feed_subtitle":"A training strategy that uses whole-video context while updating on short clips beats prior models with less memory.","key_machinery":"The load-bearing mechanism is the truncated-backpropagation loop in Algorithm 1: the bidirectional recurrent VSR model first runs a forward-only pass over the entire long video and stores hidden states for every frame; then it samples a short clip at a random start position and backpropagates through that clip alone, feeding the stored hidden states from two frames before and after the clip as context. The companion component is the refocused intra&inter-frame transformer block (RITB), which draws Query tokens from the current frame, Key/Value tokens from the current frame plus the aligned hidden states of the previous two frames, computes attention with $\\mathrm{ReLU}^2$ in place of SoftMax, and multiplies the aligned previous-frame hidden state into the feed-forward network through a refocused gated unit. This splits the cost of long-range context (one forward pass per outer iteration) from the cost of learning (many short backpropagations), which is what makes the efficiency claim work.","core_discovery":"The paper's central claim is that long-range temporal learning and short-clip training need not conflict. A recurrent VSR model absorbs long-range information during a forward-only pass over the whole video, whose hidden states are stored; backpropagation then runs only on a randomly sampled short clip, with the stored states just outside the clip serving as accurate context. Because the clip boundaries carry the same propagation states the model would use at inference, the gradient signal reflects true long-range dependencies rather than a short-window approximation. The paper further claims that this context is best used selectively: replacing SoftMax with the sparse activation $\\mathrm{ReLU}^2$ in intra- and inter-frame attention and gating the feed-forward network with the aligned previous hidden state improves the masked intra&inter-frame attention baseline by 0.24 dB on REDS4 with no added parameters, and the combined system reports 33.06 dB on REDS4 and 34.81 dB on ToS3.","pith_inferences":["A natural next test is to apply the forward-long/backward-short protocol to other recurrent low-level vision tasks, such as video deblurring or video denoising, where the paper only provides a qualitative real-world VSR example.","The paper fixes the long-context forward pass until the next outer-loop iteration; periodically refreshing it (say every $K$ short-clip updates) would map the trade-off between context staleness and compute that the paper leaves unexplored.","Because $\\mathrm{ReLU}^2$ is a sparsity prior on attention scores, the same replacement could be tested in single-image transformer super-resolution models, where suppressing low-scoring tokens may behave differently from distributing probability mass with SoftMax."],"forward_implications":["Applied unchanged to BasicVSR, BasicVSR++, and PSRT, the training strategy lifts REDS4 PSNR by 0.14, 0.10, and 0.12 dB respectively, so any recurrent VSR model can take the long-context benefit without architectural changes.","Training with 8-frame clips plus long-video context matches training with 24-frame clips, at roughly 2.5 times less training time and 2.9 times less GPU memory.","The complete LRTI-VSR model reaches 33.06 dB on REDS4 and 34.81 dB on ToS3 with 12.9 million parameters and 1.54 T FLOPs, the latter about 40 percent lower than the IART baseline it surpasses.","Replacing SoftMax with $\\mathrm{ReLU}^2$ attention and adding the refocused gated unit contribute separately (0.19 dB and 0.14 dB over the masked intra&inter-frame attention baseline), so the gains do not hinge on one monolithic design."],"supporting_citations":[{"why":"Supplies the truncated back-propagation-through-time idea that the forward-long/backward-short training strategy adapts.","marker":"[43]"},{"why":"Defines back-propagation through time, the full-sequence gradient computation whose cost the strategy avoids.","marker":"[42]"},{"why":"The BasicVSR recurrent baseline with bidirectional propagation that gains 0.14 dB REDS4 PSNR when trained with the strategy.","marker":"[5]"},{"why":"The BasicVSR++ second-order grid propagation framework on which LRTI-VSR is built and which gains 0.10 dB from the strategy.","marker":"[6]"},{"why":"The PSRT recurrent transformer whose patch-alignment module is used and which gains 0.12 dB with the strategy.","marker":"[35]"},{"why":"The masked intra&inter-frame attention block that the proposed RITB extends and the baseline it is ablated against.","marker":"[46]"},{"why":"The IART model that is the closest prior comparison, surpassed on both REDS4 and ToS3.","marker":"[44]"},{"why":"The REDS dataset supplies training sequences and the REDS4 test clips for the reported PSNR and SSIM numbers.","marker":"[29]"},{"why":"The ToS3 long-video test set used to evaluate long-range temporal information.","marker":"[10]"}],"fun_headline_variants":["Long video context, short clip training: VSR hits 33.06 dB","Train on short clips, learn from whole video: 33.06 dB VSR","ReLU-squared attention brings long-range VSR to 33.06 dB","Whole-video context without the memory cost: 33.06 dB VSR","Selective temporal focus boosts video super-res to 33.06 dB"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that hidden states from a single forward pass over a long video, computed while the model's weights are still changing, are accurate enough to serve as fixed context for backpropagation on short clips; if those bootstrapped states are too noisy early in training, the gradients are misdirected and the reported efficiency gain would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Long video context, short clip training: VSR hits 33.06 dB","Train on short clips, learn from whole video: 33.06 dB VSR","ReLU-squared attention brings long-range VSR to 33.06 dB","Whole-video context without the memory cost: 33.06 dB VSR","Selective temporal focus boosts video super-res to 33.06 dB"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1618,"prompt_tokens":933,"completion_tokens":685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":581}},"tokens_in":549,"tokens_out":685,"duration_ms":5560,"temperature":1.0,"reasoning_tokens":581,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T01:00:19.680795+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same recurrent VSR architecture on REDS4 under two protocols matched in model, clip length, sampling schedule, and iterations: the paper's one-forward-pass-per-outer-loop protocol, and a version that recomputes the long-video hidden states after every short-clip gradient update. If the recomputed-context version outperforms the paper's by a clear margin, the fixed-context assumption is the bottleneck; if the two are close, a single forward pass is sufficient context.","supporting_citations":[{"cited_title":"Gradient-based learning algorithms for recurrent networks and their computational complexity","cited_arxiv_id":null,"evidence_quote":"Supplies the truncated back-propagation-through-time idea that the forward-long/backward-short training strategy adapts."},{"cited_title":"Basicvsr: The search for essential compo- nents in video super-resolution and beyond","cited_arxiv_id":null,"evidence_quote":"The BasicVSR recurrent baseline with bidirectional propagation that gains 0.14 dB REDS4 PSNR when trained with the strategy."},{"cited_title":"Basicvsr++: Improving video super- resolution with enhanced propagation and alignment","cited_arxiv_id":null,"evidence_quote":"The BasicVSR++ second-order grid propagation framework on which LRTI-VSR is built and which gains 0.10 dB from the strategy."},{"cited_title":"Video super-resolution trans- former with masked inter&intra-frame attention","cited_arxiv_id":null,"evidence_quote":"The masked intra&inter-frame attention block that the proposed RITB extends and the baseline it is ablated against."},{"cited_title":"Enhancing Video Super-Resolution via Implicit Resampling-based Alignment","cited_arxiv_id":"2305.00163","evidence_quote":"The IART model that is the closest prior comparison, surpassed on both REDS4 and ToS3."},{"cited_title":"Ntire 2019 challenge on video deblurring and super- resolution: Dataset and study","cited_arxiv_id":null,"evidence_quote":"The REDS dataset supplies training sequences and the REDS4 test clips for the reported PSNR and SSIM numbers."},{"cited_title":"Learning temporal coherence via self- supervision for gan-based video generation","cited_arxiv_id":null,"evidence_quote":"The ToS3 long-video test set used to evaluate long-range temporal information."}],"review_version":1}