{"id":"afd4b5f6-390e-4ed7-9e56-b37ddbf541df","arxiv_id":"2412.07761","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RE-VDM adapts the frozen Stable Video Diffusion model to event-based video frame interpolation via event-conditioned ControlNet blocks and two-sided latent fusion, reporting state of the art results on BS-ERGB, HQF, and a new Clear-Motion dataset.","lead":"This paper adapts a pre-trained video diffusion model (Stable Video Diffusion) to event-based video frame interpolation by adding a trainable event-control branch and test-time start/end frame fusion. The authors report better interpolation quality and cross-camera generalization than existing frame-only and event-based baselines, including on a new self-collected test set.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative superiority may be an artifact of the evaluation protocol: baselines are VAE-coded through SVD while RE-VDM's upsampled-tile pipeline avoids that loss, and per-dataset upsampling factors are test-set tuned.","rationale":"The reader identified the load-bearing assumption as whether SVD's pretrained video prior transfers to event-camera imagery and survives the small BS-ERGB adaptation. I agree that this transfer is ultimately what the paper claims, but I find a more immediate and checkable weakness in the evaluation protocol itself. The paper's central quantitative claim rests on comparisons in Table 2. Section 4.2 explicitly applies SVD's VAE encode/decode to all model outputs, while Section 3.4 and Figure 5 show that the proposed method mitigates the VAE loss by upsampling inputs and operating on tiles. This creates a systematic asymmetry: every baseline output is degraded by a lossy autoencoder that the proposed method is specifically designed to avoid. In addition, the upsampling factors used at test time are different for different datasets without any stated validation procedure. If the upsampling factor was chosen by looking at test performance, the reported numbers are a form of test-set tuning, which contradicts the claimed zero-shot generalization. These issues are concrete, falsifiable, and directly affect the strength of the central claim. I do not argue that the method is wrong or that its components are internally inconsistent; rather, the evidence presented is not sufficient to establish that RE-VDM outperforms existing methods on unseen cameras. A re-evaluation under a matched protocol is needed. Because the method remains plausible and the underlying ideas are reasonable, the appropriate verdict is CONDITIONAL rather than REJECT or ACCEPT. My concern differs from the reader's weakest_assumption in mechanism: the reader focused on transfer failure, while I focus on evaluation bias; hence partial agreement.","tokens_in":16619,"tokens_out":5505,"duration_ms":65963,"concrete_test":"Recompute Table 2 under three protocols: (i) native outputs for all non-latent baselines with no VAE post-processing; (ii) all outputs VAE encode/decode after applying the same upsampling factor as RE-VDM for each dataset (2 for BS-ERGB/Clear-Motion, 3 for HQF); (iii) the current protocol. Also report mean and standard deviation over multiple runs or bootstrap over frames. If RE-VDM does not retain the top PSNR and SSIM on both BS-ERGB and the unseen HQF/Clear-Motion sets under protocols (i) and (ii), the claim of outperforming existing methods and generalizing across cameras is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that RE-VDM 'outperforms existing methods and generalizes across cameras far better than existing approaches' (abstract, Section 4.3). The load-bearing issue is not the internal consistency of the diffusion equations but the fairness of the comparison that supports this claim. Section 4.2 states: 'we apply VAE encoding and decoding to all model outputs.' For non-latent baselines such as CBMNet-Large, GIMM-VFI, and RIFE, this means their native outputs are passed through Stable Video Diffusion's lossy VAE before metrics are computed. Meanwhile, RE-VDM is specifically engineered to mitigate this exact loss: Section 3.4 upsampled tiles, and Figure 5 shows that VAE encode/decode PSNR rises from 21.50 at native resolution to 24.92 after 2x upsampling. Thus the comparison is asymmetric: baselines are penalized by a codec that the proposed method is designed to evade. Furthermore, the upsampling factor is chosen per test dataset (2 for BS-ERGB and Clear-Motion, 3 for HQF, Section 4.2) with no validation procedure reported, so these factors may be selected on the test sets themselves. If the reported margins over CBMNet-Large (e.g., 27.74 vs. 26.24 PSNR on BS-ERGB, 29.04 vs. 28.73 on HQF) shrink or reverse under a matched evaluation protocol, the headline generalization claim is not supported. The transfer of SVD priors is a plausible mechanism, but the empirical evidence for it is confounded by this evaluation design.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RE-VDM, a method that adapts a pre-trained video diffusion foundation model (Stable Video Diffusion) to event-based video frame interpolation. The authors keep the base diffusion model frozen and train a ControlNet-style trainable copy conditioned on a multi-stack event representation, then apply two test-time mechanisms: per-tile denoising and fusion on upsampled inputs to reduce VAE reconstruction loss, and a two-side fusion that interpolates between forward and backward denoised latents so that both start and end frames constrain the result. The model is trained on BS-ERGB and evaluated on BS-ERGB, HQF, and a newly collected Clear-Motion test set. The central claims are that RE-VDM outperforms existing frame-only, event-based, and video-diffusion baselines on PSNR and SSIM, and that it generalizes across cameras far better than prior EVFI methods.","tokens_in":16955,"tokens_out":6783,"duration_ms":64136,"significance":"If the performance and generalization claims hold, this would be an important first demonstration that large pre-trained video diffusion priors can be repurposed for event-based interpolation, potentially addressing the data-scarcity problem in EVFI. The paper's formulation is internally consistent: the training objective is a standard noise-prediction loss, the event conditioning design is plausible, and the supplementary material provides substantial implementation detail. The introduction of the Clear-Motion test sequences is a useful community resource for zero-shot evaluation. However, the evidence for the headline claims is weakened by a comparison protocol that appears to favor the proposed method in multiple ways: the VAE round-trip is applied asymmetrically, the upsampling factor is tuned per test dataset without a documented validation procedure, and recent state-of-the-art EVFI baselines are omitted. These issues are load-bearing because they directly affect whether Table 2 supports the abstract-level claim of superiority and cross-camera generalization.","major_comments":[{"comment":"The VAE round-trip applied to all model outputs does not equalize the comparison in the way claimed. For CBMNet-Large, RIFE, GIMM-VFI, and other pixel-space baselines, their native predictions are passed through SVD's VAE encoder and decoder, incurring a lossy transform that RE-VDM is explicitly engineered to mitigate through upsampled per-tile denoising (§3.4). RE-VDM's outputs are already in the VAE-decoded domain, so the VAE round-trip is not an additional distortion for the proposed method; it is intrinsic to its generation path. The statement that applying VAE encoding/decoding to all outputs 'eliminates' distribution discrepancy is therefore not supported. The reported margins, e.g., 27.74 vs. 26.24 PSNR on BS-ERGB, could be partly an artifact of this asymmetry. Please report baselines both with and without the VAE round-trip, and also report RE-VDM after an explicit additional VAE encode/decode, to show that the margins persist under a matched protocol.","section":"§4.2, Table 2"},{"comment":"The upsampling factor in Per-tile Denoising and Fusion is set per test dataset (2 for BS-ERGB and Clear-Motion, 3 for HQF) with no validation procedure described. The supplementary material, Table 5, shows that on BS-ERGB this choice changes PSNR by roughly 3 dB (24.82 at factor 1 vs. 27.74 at factor 2), so it is a strong hyperparameter. Selecting the factor after seeing each test set confounds the claim that the method 'generalizes across cameras far better than existing approaches': the reported cross-dataset gains may reflect test-set-specific tuning rather than the transferable video prior. Please describe how the upsampling factor was chosen, ideally using a held-out validation split, and report sensitivity across all three datasets.","section":"§4.2"},{"comment":"Only CBMNet-Large is included as an event-based baseline, while TimeLens++ [42] and TimeLens-XL [28] are cited in §2 as state-of-the-art EVFI methods and are not evaluated. Since the central claim is that RE-VDM outperforms existing methods, excluding the most recent EVFI baselines from the comparison materially weakens that claim. Please add these baselines using the same evaluation protocol, or provide a documented justification for their exclusion.","section":"§4.3, Table 2"},{"comment":"The abstract's unqualified statement that the method 'outperforms existing methods' is not supported by the LPIPS column. On all three datasets, several frame-only baselines achieve lower (better) LPIPS than RE-VDM: on BS-ERGB, GIMM-VFI-R-P has 0.08 vs. 0.12; on HQF, GIMM-VFI-R-P has 0.04 vs. 0.06; on Clear-Motion, PerVFI has 0.30 vs. 0.37. Section 4.3 correctly limits the claim to PSNR and SSIM, but the abstract and introduction should be qualified accordingly to avoid overstating the result.","section":"Abstract, §4.3, Table 2"},{"comment":"No variance or repeated evaluation is reported. RE-VDM is a stochastic diffusion process that runs 25 denoising steps from random latent initialization, so single-run metrics are insufficient to establish that the PSNR/SSIM margins are statistically reliable. Please report mean ± standard deviation over at least three seeds for the proposed method, and where applicable for stochastic baselines, so that the reader can assess the stability of the reported improvements.","section":"Table 2"}],"minor_comments":[{"comment":"The notation Wf ∈ R^F with 'W0 = 0 and W1 = 1' is ambiguous; please specify whether Wf is a vector indexed by frame position and state explicitly how intermediate frames are assigned weights.","section":"Eq. (3), §3.5"},{"comment":"The multi-stack event representation is described qualitatively, but the number of stacks M and the exact event counts per stack are not specified; a precise definition would aid reproducibility.","section":"§3.3, Figure 4"},{"comment":"TimeRewind [10] is cited as an image-and-events video diffusion approach from the same group but is not compared; since it is closely related, the paper should either include it in the experiments or explicitly state how RE-VDM differs from it.","section":"§2, Related Work"},{"comment":"The run time and memory comparison is reported for an RTX 4090, but the training setup in §4.1 uses RTX A6000 GPUs; please clarify whether the inference measurements were all obtained on the same hardware and resolution, since Table 6 lists different resolutions across the main text.","section":"Table 6, §14 (supplementary)"},{"comment":"The PSNR values for VAE encoding/decoding at native and 2x resolutions do not state whether the comparison is performed after downsampling back to the original resolution; please clarify the evaluation resolution for these numbers.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The evaluation asymmetry in §4.2 is the main obstacle to acceptance: the comparison protocol may systematically favor RE-VDM through a mixture of VAE round-trip treatment and per-dataset upsampling tuning. The missing TimeLens baselines should also be addressed. If the authors can show the margins persist under a matched protocol, the paper would be a solid contribution to the EVFI community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper adapts Stable Video Diffusion to event-based frame interpolation via a frozen SVD backbone plus a trainable ControlNet-style event-conditioned copy, and adds two test-time tricks: upsampled per-tile denoising/fusion and two-side fusion that runs the denoiser from both start and end frames. That combination, plus a new Clear-Motion test set, is new. The work is timely, and the qualitative results on large-motion, unseen-camera sequences look genuinely stronger than frame-only baselines and CBMNet. The authors are honest about limitations and report run-time/memory/parameters.\n\nThe soft spots are in the evaluation. The biggest one: to compare latent and non-latent models, they VAE-encode/decode all outputs, but their own Figure 5 shows this operation is lossy (21.5 dB PSNR on native-resolution input) and that upsampling mitigates that loss. The proposed method is specifically engineered to avoid that loss, so the common evaluation procedure does not affect all methods equally. The reported margins over CBMNet could shrink or reverse under a matched protocol. Second, the only event-based baseline is CBMNet-Large; TimeLens++ and TimeLens-XL are cited but not run. Third, the upsampling factor is chosen per test set (2 for BS-ERGB/Clear-Motion, 3 for HQF) with no validation procedure, so it is effectively test-set tuned. Fourth, the \"first adaptation\" claim is complicated by their own TimeRewind paper, which is cited but not compared.\n\nNone of this makes the method implausible. The internal logic is fine, and the two-side fusion and multi-stack event representation are sensible. But the central claim—\"outperforms existing methods and generalizes far better\"—is currently supported by evidence with a few confounds. A serious referee should ask for: native-space metrics without VAE coding for pixel-space baselines, or an honest analysis of how the VAE affects each method; a broader set of event-based baselines; validation-based selection of upsampling factors; and a direct comparison with TimeRewind.\n\nWho is this for? Anyone working on event-based video interpolation or on adapting video diffusion models to low-data, high-fidelity tasks. It deserves peer review; the idea is worth engaging with even though the numbers are not yet trustworthy.\n\nRecommendation: send to review, with expectation of major revision.","headline":"A plausible and timely way to adapt SVD for event-based interpolation, but the headline SOTA claim rests on an evaluation protocol that may be flattering to the method.","tokens_in":17539,"tokens_out":3849,"would_cite":true,"duration_ms":34260,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen pre-trained video diffusion models can be adapted to event-based frame interpolation and outperform specialized methods on unseen cameras.","keywords":["event-based video frame interpolation","video diffusion models","test-time optimization","event camera","zero-shot generalization","cross-modal conditioning","latent diffusion","video interpolation"],"falsifier":"Train RE-VDM with the paper's exact protocol but replace BS-ERGB with a different small paired event-frame dataset (for example HQF), then evaluate zero-shot on BS-ERGB and Clear-Motion; if PSNR and SSIM fall back toward or below the CBMNet-Large baseline, the claim that the pretrained prior is the source of generalization is weakened. Equivalently, run the released checkpoint on sequences from an event camera with a different sensor resolution and lens than the one used to collect Clear-Motion; if accuracy degrades to the level of the specialized baselines, the 'generalizes across cameras' claim fails.","tokens_in":16391,"feed_emoji":"🎥","tokens_out":9201,"duration_ms":69986,"temperature":0.7,"pith_summary":"Event-based video frame interpolation (EVFI) has been held back by a scarcity of paired event-frame training data, so its specialized networks generalize poorly to new cameras and motion patterns. This paper argues that the remedy is to borrow the visual and temporal priors of a large pre-trained video diffusion model rather than training a task-specific model from scratch. The proposed method, RE-VDM, keeps the diffusion backbone frozen, adds a small trainable event-conditioning adapter trained on a single paired dataset, and uses two test-time optimizations to achieve high-fidelity interpolation. On real-world datasets, including a new Clear-Motion sequence set, the method reports the highest PSNR and SSIM among frame-only, event-based, and video-diffusion baselines, with the largest gains on unseen data. If correct, this shows that EVFI can ride on the scale of internet video data instead of building its own.","feed_headline":"Video diffusion prior beats custom models on unseen cameras","feed_subtitle":"A small adapter makes a pre-trained video diffusion model interpolate large-motion frames on new cameras.","key_machinery":"The load-bearing mechanism is the frozen-backbone plus trainable-copy adapter, following a control-conditioning design: a copy of a subset of the denoiser's blocks receives the concatenation of image latent and event latent, and its output is added as a residual to the frozen network's output, so the pre-trained video prior is never overwritten. The event condition itself is a multi-stack histogram representation that accumulates events backward in time from the target frame with halving counts per stack, capturing both fast and slow motion in one frame-like tensor. Two test-time identities complete the method: per-tile denoising and fusion combines denoised latents of overlapping upsampled tiles through weighted accumulation, $\\tilde{Z}_{t-1}=\\sum_i \\frac{W_i}{\\sum_j W_j}\\otimes Z^i_{t-1}$, to maintain resolution and control precision; and two-side fusion blends forward and backward denoised latents as $\\tilde{Z}_{t-1}=W_f\\otimes Z^s_{t-1}+(1-W_f)\\otimes\\mathrm{flip}(Z^e_{t-1})$, with $W_f$ linearly interpolating from 0 to 1 across frames. This two-side fusion is what converts the diffusion model from extrapolation to interpolation without any additional training.","core_discovery":"The paper's central claim is that a pre-trained video diffusion foundation model, left frozen, can be turned into a state-of-the-art event-based frame interpolator by training only a small residual adapter and then applying two test-time procedures. The adapter—an event encoder plus a trainable copy of a subset of the denoiser blocks—learns to convert a multi-stack event representation into a conditioning latent that steers the frozen model's denoising. At inference, per-tile denoising with upsampled inputs preserves fine detail that the latent VAE would otherwise destroy, and two-side fusion runs denoising from the start frame with forward events and from the end frame with backward events, blending the two predicted latents with linearly increasing weights so every interpolated frame is anchored to both endpoints. Trained only on the BS-ERGB dataset, the method is evaluated zero-shot on HQF and on the paper's new Clear-Motion sequences, and it outperforms all tested baselines in PSNR and SSIM, including the specialized event-based CBMNet-Large trained on the same data.","pith_inferences":["Looking beyond the paper, the same frozen-backbone plus residual-adapter recipe could be applied to other event-guided restoration tasks, such as event deblurring or event-to-video reconstruction, because the adapter only learns how to inject the event condition, not the video content prior.","The two-side fusion idea is not specific to events: any per-frame motion cue that can be encoded as a conditioning tensor could steer a frozen video diffusion model to interpolate, so the architecture is a template for other sensor modalities.","A direct stress test would be to train the adapter on a different paired dataset (for example HQF instead of BS-ERGB) and evaluate zero-shot on BS-ERGB; this would clarify how much of the generalization comes from the prior versus the choice of training distribution.","The paper's runtime table shows the method is orders of magnitude slower than classical EVFI; a natural next step, not taken in the paper, is to distill the adapter into fewer denoising steps or into a direct regression network so the generalization benefit survives at interactive speeds."],"forward_implications":["If the central claim holds, EVFI no longer needs large paired collections: a small adapter trained on one dataset inherits the generalization of an internet-scale video prior.","New generations of video foundation models can be dropped into the same adapter, so progress in video generation should directly raise event-interpolation quality.","The reported gains on 11-15 skip (large-motion) intervals imply the prior can resolve motion ambiguity that frame-only methods cannot, making high-frame-rate reconstruction from low-rate cameras more practical.","The Clear-Motion test set, which the paper introduces solely for zero-shot testing, provides a benchmark where cross-camera generalization can be measured directly."],"supporting_citations":[{"why":"Supplies the frozen latent video diffusion backbone that the whole method adapts.","marker":"[7]"},{"why":"Establishes the frozen-network-plus-trainable-copy control scheme adopted for event conditioning.","marker":"[49]"},{"why":"Supplies the tile-based denoising and fusion method used to preserve high-resolution detail.","marker":"[2]"},{"why":"Provides the test-time optimization idea for running pre-trained video diffusion models under motion control, extended here to two-side fusion.","marker":"[11]"},{"why":"The event-based interpolation baseline trained on BS-ERGB that RE-VDM is compared against on the same data.","marker":"[21]"},{"why":"Provides the BS-ERGB dataset used to train the adapter.","marker":"[41]"},{"why":"Provides the HQF dataset used for cross-dataset zero-shot evaluation.","marker":"[39]"}],"fun_headline_variants":["Event-based video interpolation via frozen diffusion prior","Frozen diffusion adapts to event video with tiny adapter","Zero-shot event video interpolation from pretrained diffusion","Diffusion prior generalizes across cameras for event interpolation","Small adapter turns diffusion model into event interpolator"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire result rests on the empirical premise that the visual and temporal priors of a large pre-trained video diffusion model transfer to the noisy, high-dynamic-range appearance and unusual motion patterns of event-camera footage after training on only the BS-ERGB dataset.","fun_headline_variants_meta":{"raw":{"variants":["Event-based video interpolation via frozen diffusion prior","Frozen diffusion adapts to event video with tiny adapter","Zero-shot event video interpolation from pretrained diffusion","Diffusion prior generalizes across cameras for event interpolation","Small adapter turns diffusion model into event interpolator"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1267,"prompt_tokens":910,"completion_tokens":357,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":284}},"tokens_in":526,"tokens_out":357,"duration_ms":3621,"temperature":1.0,"reasoning_tokens":284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:30:35.632673+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train RE-VDM with the paper's exact protocol but replace BS-ERGB with a different small paired event-frame dataset (for example HQF), then evaluate zero-shot on BS-ERGB and Clear-Motion; if PSNR and SSIM fall back toward or below the CBMNet-Large baseline, the claim that the pretrained prior is the source of generalization is weakened. Equivalently, run the released checkpoint on sequences from an event camera with a different sensor resolution and lens than the one used to collect Clear-Motion; if accuracy degrades to the level of the specialized baselines, the 'generalizes across cameras' claim fails.","supporting_citations":[{"cited_title":"Multidiffusion: Fusing diffusion paths for controlled image generation","cited_arxiv_id":null,"evidence_quote":"Supplies the tile-based denoising and fusion method used to preserve high-resolution detail."},{"cited_title":"Explo- rative inbetweening of time and space","cited_arxiv_id":null,"evidence_quote":"Provides the test-time optimization idea for running pre-trained video diffusion models under motion control, extended here to two-side fusion."},{"cited_title":"Event-based video frame interpolation with cross- modal asymmetric bidirectional motion fields","cited_arxiv_id":null,"evidence_quote":"The event-based interpolation baseline trained on BS-ERGB that RE-VDM is compared against on the same data."},{"cited_title":"Time lens: Event-based video frame interpolation","cited_arxiv_id":null,"evidence_quote":"Provides the BS-ERGB dataset used to train the adapter."},{"cited_title":"Reducing the sim-to-real gap for event cameras","cited_arxiv_id":null,"evidence_quote":"Provides the HQF dataset used for cross-dataset zero-shot evaluation."}],"review_version":1}