Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Frozen pre-trained video diffusion models can be adapted to event-based frame interpolation and outperform specialized methods on unseen cameras.

desk verdict A plausible and timely way to adapt SVD for event-based interpolation, but the headline SOTA claim rests on an evaluation protocol that may be flattering to the method. read the letter →

arxiv 2412.07761 v2 pith:MIXRMKND submitted 2024-12-10 cs.CV

Jingxi Chen , Brandon Y. Feng , Haoming Cai , Tianfu Wang , Levi Burner , Dehao Yuan , Cornelia Fermuller , Christopher A. Metzler
show 1 more author
Yiannis Aloimonos
This is my paper · ORCID
classification cs.CV
keywords event-basedvideoframeinterpolationdiffusionmodelstest-timeoptimizationeventcamerazero-shotgeneralizationcross-modalconditioninglatent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Event-based video frame interpolation (EVFI) has been held back by a scarcity of paired event-frame training data, so its specialized networks generalize poorly to new cameras and motion patterns. This paper argues that the remedy is to borrow the visual and temporal priors of a large pre-trained video diffusion model rather than training a task-specific model from scratch. The proposed method, RE-VDM, keeps the diffusion backbone frozen, adds a small trainable event-conditioning adapter trained on a single paired dataset, and uses two test-time optimizations to achieve high-fidelity interpolation. On real-world datasets, including a new Clear-Motion sequence set, the method reports the highest PSNR and SSIM among frame-only, event-based, and video-diffusion baselines, with the largest gains on unseen data. If correct, this shows that EVFI can ride on the scale of internet video data instead of building its own.

What carries the argument

The load-bearing mechanism is the frozen-backbone plus trainable-copy adapter, following a control-conditioning design: a copy of a subset of the denoiser's blocks receives the concatenation of image latent and event latent, and its output is added as a residual to the frozen network's output, so the pre-trained video prior is never overwritten. The event condition itself is a multi-stack histogram representation that accumulates events backward in time from the target frame with halving counts per stack, capturing both fast and slow motion in one frame-like tensor. Two test-time identities complete the method: per-tile denoising and fusion combines denoised latents of overlapping upsampled tiles through weighted accumulation, $\tilde{Z}_{t-1}=\sum_i \frac{W_i}{\sum_j W_j}\otimes Z^i_{t-1}$, to maintain resolution and control precision; and two-side fusion blends forward and backward denoised latents as $\tilde{Z}_{t-1}=W_f\otimes Z^s_{t-1}+(1-W_f)\otimes\mathrm{flip}(Z^e_{t-1})$, with $W_f$ linearly interpolating from 0 to 1 across frames. This two-side fusion is what converts the diffusion model from extrapolation to interpolation without any additional training.

What would settle it

Train RE-VDM with the paper's exact protocol but replace BS-ERGB with a different small paired event-frame dataset (for example HQF), then evaluate zero-shot on BS-ERGB and Clear-Motion; if PSNR and SSIM fall back toward or below the CBMNet-Large baseline, the claim that the pretrained prior is the source of generalization is weakened. Equivalently, run the released checkpoint on sequences from an event camera with a different sensor resolution and lens than the one used to collect Clear-Motion; if accuracy degrades to the level of the specialized baselines, the 'generalizes across cameras' claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a pre-trained video diffusion foundation model, left frozen, can be turned into a state-of-the-art event-based frame interpolator by training only a small residual adapter and then applying two test-time procedures. The adapter—an event encoder plus a trainable copy of a subset of the denoiser blocks—learns to convert a multi-stack event representation into a conditioning latent that steers the frozen model's denoising. At inference, per-tile denoising with upsampled inputs preserves fine detail that the latent VAE would otherwise destroy, and two-side fusion runs denoising from the start frame with forward events and from the end frame with backward events, blending the two predicted latents with linearly increasing weights so every interpolated frame is anchored to both endpoints. Trained only on the BS-ERGB dataset, the method is evaluated zero-shot on HQF and on the paper's new Clear-Motion sequences, and it outperforms all tested baselines in PSNR and SSIM, including the specialized event-based CBMNet-Large trained on the same data.

Load-bearing premise

The entire result rests on the empirical premise that the visual and temporal priors of a large pre-trained video diffusion model transfer to the noisy, high-dynamic-range appearance and unusual motion patterns of event-camera footage after training on only the BS-ERGB dataset.

Editorial extensions

If this is right

  • If the central claim holds, EVFI no longer needs large paired collections: a small adapter trained on one dataset inherits the generalization of an internet-scale video prior.
  • New generations of video foundation models can be dropped into the same adapter, so progress in video generation should directly raise event-interpolation quality.
  • The reported gains on 11-15 skip (large-motion) intervals imply the prior can resolve motion ambiguity that frame-only methods cannot, making high-frame-rate reconstruction from low-rate cameras more practical.
  • The Clear-Motion test set, which the paper introduces solely for zero-shot testing, provides a benchmark where cross-camera generalization can be measured directly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Looking beyond the paper, the same frozen-backbone plus residual-adapter recipe could be applied to other event-guided restoration tasks, such as event deblurring or event-to-video reconstruction, because the adapter only learns how to inject the event condition, not the video content prior.
  • The two-side fusion idea is not specific to events: any per-frame motion cue that can be encoded as a conditioning tensor could steer a frozen video diffusion model to interpolate, so the architecture is a template for other sensor modalities.
  • A direct stress test would be to train the adapter on a different paired dataset (for example HQF instead of BS-ERGB) and evaluate zero-shot on BS-ERGB; this would clarify how much of the generalization comes from the prior versus the choice of training distribution.
  • The paper's runtime table shows the method is orders of magnitude slower than classical EVFI; a natural next step, not taken in the paper, is to distill the adapter into fewer denoising steps or into a direct regression network so the generalization benefit survives at interactive speeds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes RE-VDM, a method that adapts a pre-trained video diffusion foundation model (Stable Video Diffusion) to event-based video frame interpolation. The authors keep the base diffusion model frozen and train a ControlNet-style trainable copy conditioned on a multi-stack event representation, then apply two test-time mechanisms: per-tile denoising and fusion on upsampled inputs to reduce VAE reconstruction loss, and a two-side fusion that interpolates between forward and backward denoised latents so that both start and end frames constrain the result. The model is trained on BS-ERGB and evaluated on BS-ERGB, HQF, and a newly collected Clear-Motion test set. The central claims are that RE-VDM outperforms existing frame-only, event-based, and video-diffusion baselines on PSNR and SSIM, and that it generalizes across cameras far better than prior EVFI methods.

Significance. If the performance and generalization claims hold, this would be an important first demonstration that large pre-trained video diffusion priors can be repurposed for event-based interpolation, potentially addressing the data-scarcity problem in EVFI. The paper's formulation is internally consistent: the training objective is a standard noise-prediction loss, the event conditioning design is plausible, and the supplementary material provides substantial implementation detail. The introduction of the Clear-Motion test sequences is a useful community resource for zero-shot evaluation. However, the evidence for the headline claims is weakened by a comparison protocol that appears to favor the proposed method in multiple ways: the VAE round-trip is applied asymmetrically, the upsampling factor is tuned per test dataset without a documented validation procedure, and recent state-of-the-art EVFI baselines are omitted. These issues are load-bearing because they directly affect whether Table 2 supports the abstract-level claim of superiority and cross-camera generalization.

major comments (5)
  1. [§4.2, Table 2] The VAE round-trip applied to all model outputs does not equalize the comparison in the way claimed. For CBMNet-Large, RIFE, GIMM-VFI, and other pixel-space baselines, their native predictions are passed through SVD's VAE encoder and decoder, incurring a lossy transform that RE-VDM is explicitly engineered to mitigate through upsampled per-tile denoising (§3.4). RE-VDM's outputs are already in the VAE-decoded domain, so the VAE round-trip is not an additional distortion for the proposed method; it is intrinsic to its generation path. The statement that applying VAE encoding/decoding to all outputs 'eliminates' distribution discrepancy is therefore not supported. The reported margins, e.g., 27.74 vs. 26.24 PSNR on BS-ERGB, could be partly an artifact of this asymmetry. Please report baselines both with and without the VAE round-trip, and also report RE-VDM after an explicit additional VAE encode/decode, to show that the margins persist under a matched protocol.
  2. [§4.2] The upsampling factor in Per-tile Denoising and Fusion is set per test dataset (2 for BS-ERGB and Clear-Motion, 3 for HQF) with no validation procedure described. The supplementary material, Table 5, shows that on BS-ERGB this choice changes PSNR by roughly 3 dB (24.82 at factor 1 vs. 27.74 at factor 2), so it is a strong hyperparameter. Selecting the factor after seeing each test set confounds the claim that the method 'generalizes across cameras far better than existing approaches': the reported cross-dataset gains may reflect test-set-specific tuning rather than the transferable video prior. Please describe how the upsampling factor was chosen, ideally using a held-out validation split, and report sensitivity across all three datasets.
  3. [§4.3, Table 2] Only CBMNet-Large is included as an event-based baseline, while TimeLens++ [42] and TimeLens-XL [28] are cited in §2 as state-of-the-art EVFI methods and are not evaluated. Since the central claim is that RE-VDM outperforms existing methods, excluding the most recent EVFI baselines from the comparison materially weakens that claim. Please add these baselines using the same evaluation protocol, or provide a documented justification for their exclusion.
  4. [Abstract, §4.3, Table 2] The abstract's unqualified statement that the method 'outperforms existing methods' is not supported by the LPIPS column. On all three datasets, several frame-only baselines achieve lower (better) LPIPS than RE-VDM: on BS-ERGB, GIMM-VFI-R-P has 0.08 vs. 0.12; on HQF, GIMM-VFI-R-P has 0.04 vs. 0.06; on Clear-Motion, PerVFI has 0.30 vs. 0.37. Section 4.3 correctly limits the claim to PSNR and SSIM, but the abstract and introduction should be qualified accordingly to avoid overstating the result.
  5. [Table 2] No variance or repeated evaluation is reported. RE-VDM is a stochastic diffusion process that runs 25 denoising steps from random latent initialization, so single-run metrics are insufficient to establish that the PSNR/SSIM margins are statistically reliable. Please report mean ± standard deviation over at least three seeds for the proposed method, and where applicable for stochastic baselines, so that the reader can assess the stability of the reported improvements.
minor comments (5)
  1. [Eq. (3), §3.5] The notation Wf ∈ R^F with 'W0 = 0 and W1 = 1' is ambiguous; please specify whether Wf is a vector indexed by frame position and state explicitly how intermediate frames are assigned weights.
  2. [§3.3, Figure 4] The multi-stack event representation is described qualitatively, but the number of stacks M and the exact event counts per stack are not specified; a precise definition would aid reproducibility.
  3. [§2, Related Work] TimeRewind [10] is cited as an image-and-events video diffusion approach from the same group but is not compared; since it is closely related, the paper should either include it in the experiments or explicitly state how RE-VDM differs from it.
  4. [Table 6, §14 (supplementary)] The run time and memory comparison is reported for an RTX 4090, but the training setup in §4.1 uses RTX A6000 GPUs; please clarify whether the inference measurements were all obtained on the same hardware and resolution, since Table 6 lists different resolutions across the main text.
  5. [Figure 5] The PSNR values for VAE encoding/decoding at native and 2x resolutions do not state whether the comparison is performed after downsampling back to the original resolution; please clarify the evaluation resolution for these numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the method is measured against external datasets and baselines; the test-set-tuned upsampling factor is a protocol concern, not a definitional circularity.

full rationale

Walking the derivation chain: RE-VDM introduces event control through a ControlNet-style frozen-backbone plus trainable residual copy (Sec. 3.2), trained with the standard noise-prediction MSE objective on BS-ERGB latents. No reported metric or baseline output is fed back as a training input, and the fusion equations (Eq. 2 and Eq. 3) combine the model's own denoised latents rather than inverting any fitted quantity. The central empirical claim is evaluated against external corpora: BS-ERGB, HQF, and a self-collected Clear-Motion set explicitly used only for testing, against publicly available baseline checkpoints. The only self-citation, TimeRewind [10], appears as background for the event-plus-image line of work and is not the justification for any technical choice; the two-side fusion idea is attributed to external work [11] and tiled denoising to MultiDiffusion [2]. The per-dataset upsampling factors (2 for BS-ERGB and Clear-Motion, 3 for HQF) are hyperparameters selected without a reported validation split, so the reported margins may be optimistic, and applying SVD's VAE to all outputs burdens non-latent baselines asymmetrically; however, these are evaluation-protocol and selection-bias concerns, not circularity, because the reported PSNR/SSIM values are not equal to the fitted factors by construction and no load-bearing step reduces to its own input. Therefore the analysis finds no circular step and assigns score 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities; RE-VDM is a model architecture and test-time procedure, not a postulated entity. The main burdens are empirical: transfer of the SVD prior, the event representation, and the test-time hyperparameters.

free parameters (4)
  • Per-dataset upsampling factor = 2 (BS-ERGB, Clear-Motion), 3 (HQF)
    Selected per test dataset with Supplementary Table 5 showing tuning on BS-ERGB test metrics; no validation split is described.
  • Tile size and overlap ratio = 512x320 tiles, overlap ratio 0.1
    Chosen at test time without ablation; affects detail preservation and event control accuracy.
  • Number of denoising steps = 25
    Set for all sampling at test time; no ablation on step count or effect on interpolation quality.
  • Event representation stack count = 3 stacks (implied by Figure 4)
    Multi-stack event representation hyperparameters are not fully specified, yet they determine how motion information is fed to the model.
assumptions (5)
  • domain assumption Events provide reliable motion guidance between frames.
    Used throughout Section 3 as the basis for event conditioning; if event-to-motion mapping is noisy or misaligned, interpolation quality degrades.
  • domain assumption Stable Video Diffusion's pretrained prior transfers to event-camera imagery.
    Assumed in Section 1 and Section 3.2; validated only empirically on three real-world datasets, with no formal derivation.
  • domain assumption Training only on BS-ERGB with 3 and 11 skipped frames generalizes to 15-skip evaluation and new cameras.
    Section 13 states training on BS-ERGB only; the Clear-Motion evaluation uses 15 skips, far beyond the training skip count.
  • domain assumption Latent VAE encoding/decoding loss can be adequately mitigated by upsampling and tiling.
    Central motivation for Per-tile Denoising and Fusion in Section 3.4 and Figure 5; depends on the VAE's error characteristics.
  • ad hoc to paper Linear weight fusion between forward and backward latents yields correct temporal interpolation.
    Equation 3 sets W0=0 and W1=1 with linear interpolation; no derivation or independent validation is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation." pith.science (2026). https://pith.science/paper/MIXRMKND

@misc{pith2026241207761,
  author       = {Pith},
  title        = {Pith review of: Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MIXRMKND}},
  note         = {Machine review of arXiv:2412.07761}
}
read the original abstract

Video Frame Interpolation aims to recover realistic missing frames between observed frames, generating a high-frame-rate video from a low-frame-rate video. However, without additional guidance, the large motion between frames makes this problem ill-posed. Event-based Video Frame Interpolation (EVFI) addresses this challenge by using sparse, high-temporal-resolution event measurements as motion guidance. This guidance allows EVFI methods to significantly outperform frame-only methods. However, to date, EVFI methods have relied on a limited set of paired event-frame training data, severely limiting their performance and generalization capabilities. In this work, we overcome the limited data challenge by adapting pre-trained video diffusion models trained on internet-scale datasets to EVFI. We experimentally validate our approach on real-world EVFI datasets, including a new one that we introduce. Our method outperforms existing methods and generalizes across cameras far better than existing approaches.

Figures

Figures reproduced from arXiv: 2412.07761 by the authors.

Figure 1
Figure 1. We compare our proposed approach RE-VDM, which adapts a pre-trained video diffusion model for event-based video frame [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of our proposed approach RE-VDM: for [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Our multi-stack event representation is illustrated as [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (12 more)
Figure 6
Figure 6. Figure 6: The Per-tile Denoising and Fusion process is a test [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: The qualitative comparison on the Clear-Motion sequence Texture_Box, featuring translation and rotation of a textured box, [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The qualitative comparison on the BS-ERGB test sequences, featuring fast motion of a hand and a rising, spinning ball, shows that [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: An example illustrating the difference between event [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 11
Figure 11. Figure 11: An illustration of our capture setup for the Clear-Motion [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 13
Figure 13. Figure 13: An illustration showcasing the qualitative impact of [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Additional baseline results on the Clear-Motion sequence Paper_Waving. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Additional baseline results on the BS-ERGB sequence as presented in the main paper. [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: An illustration showcasing the qualitative comparison on the Clear-Motion sequence Camera_Far, which involves large camera [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 17
Figure 17. Figure 17: An illustration showcasing the qualitative comparison on the Clear-Motion sequence Camera_Close, which involves large camera [PITH_FULL_IMAGE:figures/full_fig_p017_17.png]
Figure 18
Figure 18. Figure 18: An illustration showcasing the qualitative comparison on the Clear-Motion sequence Checkerboard_Planar, which involves large [PITH_FULL_IMAGE:figures/full_fig_p018_18.png]
Figure 19
Figure 19. Figure 19: An illustration showcasing the qualitative comparison on the HQF dataset for the sequence poster_pillar_1, involving moving [PITH_FULL_IMAGE:figures/full_fig_p018_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning Normal Flow Directly From Event Neighborhoods

    cs.CV 2024-12 reject novelty 6.0 of 10

    A point-based network learns per-event normal flow from raw event camera data and, with IMU data, estimates egomotion; it transfers across datasets better than frame-based optical flow methods.

Reference graph

Works this paper leans on

67 extracted references · 47 canonical work pages · cited by 1 Pith paper

  1. [42]

    Time lens++: Event-based frame interpolation with paramet- ric non-linear flow and multi-scale fusion

    Stepan Tulyakov, Alfredo Bochicchio, Daniel Gehrig, Sta- matios Georgoulis, Yuanyou Li, and Davide Scaramuzza. Time lens++: Event-based frame interpolation with paramet- ric non-linear flow and multi-scale fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17755–17764, 2022. 2, 3

    work page 2022

  2. [28]

    Timelens-xl: Real-time event-based video frame interpolation with large motion

    Yongrui Ma, Shi Guo, Yutian Chen, Tianfan Xue, and Jin- wei Gu. Timelens-xl: Real-time event-based video frame interpolation with large motion. In European Conference on Computer Vision, pages 178–194. Springer, 2024. 2

    work page 2024

  3. [1]

    Depth-aware video frame interpolation

    Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiy- ong Gao, and Ming-Hsuan Yang. Depth-aware video frame interpolation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3703–3712,

    work page

  4. [2]

    Multidiffusion: Fusing diffusion paths for controlled image generation

    Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. In ICML. PMLR, 2023. 3, 5

    work page 2023

  5. [3]

    Simultaneous optical flow and intensity estimation from an event camera

    Patrick Bardow, Andrew J Davison, and Stefan Leutenegger. Simultaneous optical flow and intensity estimation from an event camera. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 884–892,

    work page

  6. [4]

    Contour motion estimation for asynchronous event- driven cameras

    Francisco Barranco, Cornelia Fermüller, and Yiannis Aloi- monos. Contour motion estimation for asynchronous event- driven cameras. Proceedings of the IEEE , 102(10):1537– 1556, 2014

    work page 2014

  7. [5]

    Real-time clustering and multi-target tracking using event- based sensors

    Francisco Barranco, Cornelia Fermüller, and Eduardo Ros. Real-time clustering and multi-target tracking using event- based sensors. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5764–5769. IEEE, 2018

    work page 2018

  8. [6]

    Asynchronous frameless event-based optical flow

    Ryad Benosman, Sio-Hoi Ieng, Charles Clercq, Chiara Bar- tolozzi, and Mandyam Srinivasan. Asynchronous frameless event-based optical flow. Neural Networks, 27:32–37, 2012. 3

    work page 2012

Show all 67 references
  1. [7]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023....

    2023 arXiv

  2. [8]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In CVPR, 2023. 2, 3

    2023

  3. [9]

    Sparse-e2vid: A sparse convolutional model for event-based video reconstruction trained with real event noise

    Pablo Rodrigo Gantier Cadena, Yeqiang Qian, Chunxiang Wang, and Ming Yang. Sparse-e2vid: A sparse convolutional model for event-based video reconstruction trained with real event noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

  4. [10]

    Timerewind: Rewinding time with image-and-events video diffusion

    Jingxi Chen, Brandon Y Feng, Haoming Cai, Mingyang Xie, Christopher Metzler, Cornelia Fermuller, and Yiannis Aloi- monos. Timerewind: Rewinding time with image-and-events video diffusion. arXiv preprint arXiv:2403.13800, 2024. 3

    2024 arXiv

  5. [11]

    Explo- rative inbetweening of time and space

    Haiwen Feng, Zheng Ding, Zhihao Xia, Simon Niklaus, Vic- toria Abrevaya, Michael J Black, and Xuaner Zhang. Explo- rative inbetweening of time and space. In ECCV, 2024. 1, 3, 6

    2024

  6. [12]

    Event-based vision: A survey

    Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154...

    2020

  7. [13]

    Digital image processing

    Rafael C Gonzalez. Digital image processing. Pearson edu- cation india, 2009. 6

    2009

  8. [14]

    Generalizable implicit motion modeling for video frame interpolation

    Zujin Guo, Wei Li, and Chen Change Loy. Generalizable implicit motion modeling for video frame interpolation. Ad- vances in Neural Information Processing Systems, 37:63747– 63770, 2024. 1, 3, 6

    2024

  9. [15]

    Microsaccade-inspired event camera for robotics

    Botao He, Ze Wang, Yuan Zhou, Jingxi Chen, Chahat Deep Singh, Haojia Li, Yuman Gao, Shaojie Shen, Kaiwei Wang, Yanjun Cao, et al. Microsaccade-inspired event camera for robotics. Science Robotics, 9(90):eadj8124, 2024. 3

    2024

  10. [16]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 3

    2020

  11. [17]

    Video diffusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. 2022. 2, 3

    2022

  12. [18]

    Real-time intermediate flow estimation for video frame interpolation

    Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In European Conference on Com- puter Vision, pages 624–642. Springer, 2022. 3, 6

    2022

  13. [19]

    Dginstyle: Domain-generalizable semantic segmentation with image dif- fusion models and stylized semantic control

    Yuru Jia, Lukas Hoyer, Shengyu Huang, Tianfu Wang, Luc Van Gool, Konrad Schindler, and Anton Obukhov. Dginstyle: Domain-generalizable semantic segmentation with image dif- fusion models and stylized semantic control. In ECCV, 2024. 3

    2024

  14. [20]

    Super slomo: High quality estimation of multiple intermediate frames for video interpolation

    Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9...

  15. [21]

    Event-based video frame interpolation with cross- modal asymmetric bidirectional motion fields

    Taewoo Kim, Yujeong Chae, Hyun-Kurl Jang, and Kuk-Jin Yoon. Event-based video frame interpolation with cross- modal asymmetric bidirectional motion fields. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18032–18042, 2023. 1, 2, 6, 14

    2023

  16. [22]

    Ifrnet: Intermediate feature refine network for efficient frame interpolation

    Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang, and Jie Yang. Ifrnet: Intermediate feature refine network for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa...

    1969

  17. [23]

    Sora: A review on background, technology, limitations, and opportunities of large vision models

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jian- feng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024. 2

    2024 arXiv

  18. [24]

    Fixing weight decay regularization in adam

    Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5,

    arXiv

  19. [25]

    Video frame interpolation with transformer

    Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia. Video frame interpolation with transformer. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3532–3542, 2022. 3

    2022

  20. [26]

    Hr- inr: continuous space-time video super-resolution via event camera

    Yunfan Lu, Zipeng Wang, Yusheng Wang, and Hui Xiong. Hr- inr: continuous space-time video super-resolution via event camera. arXiv preprint arXiv:2405.13389, 2024. 4

    2024

  21. [27]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, pages 11461–11471, 2022. 3

    2022

  22. [29]

    Event-based moving object detection and tracking

    Anton Mitrokhin, Cornelia Fermüller, Chethan Paramesh- wara, and Yiannis Aloimonos. Event-based moving object detection and tracking. In 2018 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pages 1–9. IEEE, 2018. 3

    2018

  23. [30]

    Stereo depth from events cameras: Concen- trate and focus on the future

    Yeongwoo Nam, Mohammad Mostafavi, Kuk-Jin Yoon, and Jonghyun Choi. Stereo depth from events cameras: Concen- trate and focus on the future. InProceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, 2022. 4

    2022

  24. [31]

    Asymmetric bilateral motion estimation for video frame interpolation

    Junheum Park, Chul Lee, and Chang-Su Kim. Asymmetric bilateral motion estimation for video frame interpolation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14539–14548, 2021. 3

    2021

  25. [32]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  26. [33]

    High speed and high dynamic range video with an event camera

    Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. High speed and high dynamic range video with an event camera. IEEE transactions on pattern analysis and machine intelligence, 43(6):1964–1980, 2019. 3

    1964

  27. [34]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 3

    2022

  28. [35]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven gen- eration

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven gen- eration. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages ...

  29. [36]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 3

    2021 arXiv

  30. [37]

    Codedevents: optimal point-spread-function engineering for 3d-tracking with event cameras

    Sachin Shah, Matthew A Chan, Haoming Cai, Jingxi Chen, Sakshum Kulshrestha, Chahat Deep Singh, Yiannis Aloi- monos, and Christopher A Metzler. Codedevents: optimal point-spread-function engineering for 3d-tracking with event cameras. In Proceedings of the IEEE/CVF conference o...

  31. [38]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021. 3

    2021

  32. [39]

    Reducing the sim-to-real gap for event cameras

    Timo Stoffregen, Cedric Scheerlinck, Davide Scaramuzza, Tom Drummond, Nick Barnes, Lindsay Kleeman, and Robert Mahony. Reducing the sim-to-real gap for event cameras. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV...

    2020

  33. [40]

    Event-based frame interpolation with ad-hoc deblurring

    Lei Sun, Christos Sakaridis, Jingyun Liang, Peng Sun, Jiezhang Cao, Kai Zhang, Qi Jiang, Kaiwei Wang, and Luc Van Gool. Event-based frame interpolation with ad-hoc deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18043– 18...

    2023

  34. [41]

    Time lens: Event-based video frame interpolation

    Stepan Tulyakov, Daniel Gehrig, Stamatios Georgoulis, Julius Erbach, Mathias Gehrig, Yuanyou Li, and Davide Scaramuzza. Time lens: Event-based video frame interpolation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16155–16164, ...

    2021

  35. [43]

    Con- sistencyˆ 2: Consistent and fast 3d painting with latent con- sistency models

    Tianfu Wang, Anton Obukhov, and Konrad Schindler. Con- sistencyˆ 2: Consistent and fast 3d painting with latent con- sistency models. arXiv preprint arXiv:2406.11202 , 2024. 3

    2024 arXiv

  36. [44]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 6

    2004

  37. [45]

    Perception-oriented video frame interpolation via asymmetric blending

    Guangyang Wu, Xin Tao, Changlin Li, Wenyi Wang, Xiao- hong Liu, and Qingqing Zheng. Perception-oriented video frame interpolation via asymmetric blending. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2753–2762, 2024. 3, 6

    2024

  38. [46]

    Dynamicrafter: Animating open- domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open- domain images with video diffusion priors. In European Conference on Computer Vision, pages 399–417. Springer,

  39. [47]

    Learn- ing normal flow directly from event neighborhoods

    Dehao Yuan, Levi Burner, Jiayi Wu, Minghui Liu, Jingxi Chen, Yiannis Aloimonos, and Cornelia Fermüller. Learn- ing normal flow directly from event neighborhoods. arXiv preprint arXiv:2412.11284, 2024. 3

    2024 arXiv

  40. [48]

    Extracting motion and appearance via inter-frame attention for efficient video frame interpolation

    Guozhen Zhang, Yuhan Zhu, Haonan Wang, Youxin Chen, Gangshan Wu, and Limin Wang. Extracting motion and appearance via inter-frame attention for efficient video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5682–5692,

  41. [49]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. 2, 3, 4

    2023

  42. [50]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 6

    2018

  43. [51]

    Unifying motion deblurring and frame interpolation with events

    Xiang Zhang and Lei Yu. Unifying motion deblurring and frame interpolation with events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17765–17774, 2022. 2, 3

    2022

  44. [52]

    Mimicmotion: High-quality human motion video generation with confidence- aware pose guidance

    Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mimicmotion: High-quality human motion video generation with confidence- aware pose guidance. arXiv preprint arXiv:2406.19680, 2024. 3

    2024 arXiv

  45. [53]

    Clearer frames, anytime: Re- solving velocity ambiguity in video frame interpolation

    Zhihang Zhong, Gurunandan Krishnan, Xiao Sun, Yu Qiao, Sizhuo Ma, and Jian Wang. Clearer frames, anytime: Re- solving velocity ambiguity in video frame interpolation. In European Conference on Computer Vision, pages 346–363. Springer, 2024. 3, 6

    2024

  46. [54]

    Ev-flownet: Self-supervised optical flow estimation for event-based cameras

    Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. Ev-flownet: Self-supervised optical flow estimation for event-based cameras. arXiv preprint arXiv:1802.06898, 2018. 3

    2018 arXiv

  47. [55]

    Unsupervised event-based learning of optical flow, depth, and egomotion

    Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 989–997, 2019. 3 Repurposing Pre-trained ...

    2019

  48. [56]

    Event Camera and Video Frame Interpolation 2 2.2

    Related Work 2 2.1. Event Camera and Video Frame Interpolation 2 2.2. Diffusion and Video Diffusion Models . . . . 3 2.3. Controllable Diffusion Generation Through Fine-Tuning . . . . . . . . . . . . . . . . . 3

  49. [57]

    Pipeline Overview

    Proposed Approach 3 3.1. Pipeline Overview . . . . . . . . . . . . . . 3 3.2. Data-Efficient Adaptation of Pre-trained Video Diffusion Models . . . . . . . . . . . 3 3.3. Events as the Control Condition . . . . . . . 4 3.4. Per-tile Denoising and Fusion . . . . . . . . 4 3.5. ...

  50. [58]

    Datasets and Implementation Details

    Experimental Results 6 4.1. Datasets and Implementation Details . . . . 6 4.2. Evaluation Strategy . . . . . . . . . . . . . . 6 4.3. Evaluation on Video Frame Interpolation . . 7 4.4. Ablation Study . . . . . . . . . . . . . . . . 7

  51. [59]

    Clear-Motion Test Sequences 13 10.1

    Video Generation Task Results 12 10 . Clear-Motion Test Sequences 13 10.1 . Event-RGB Aligned Video Capture Setup . 13 10.2 . Details of Data Sequence . . . . . . . . . . 13 11 . The Impact of Input Upsampling 14 12 . More Visual Results 14 13 . Additional Implementation Details 14

  52. [60]

    Model Run Time, Memory, and Parameter Com- parison 15

  53. [61]

    github.io/ for video results, which clearly demonstrate that our reconstructions provide superior consistency and generalization compared to other baselines

    Video Results Please refer to our project page: https://vdm-evfi. github.io/ for video results, which clearly demonstrate that our reconstructions provide superior consistency and generalization compared to other baselines

  54. [62]

    This constraint in video generation leads to error accumulation in the generated video, as shown in the last video of the website

    Video Generation Task Results As explained in the main paper, our method supports Event- based Video Generation, an extrapolation task that relies on only one frame (start or end) and events, unlike interpolation, which uses both frames. This constraint in video generation lea...

  55. [63]

    Clear-Motion Test Sequences To robustly evaluate the zero-shot generalization perfor- mance of all models on unseen real-world event-based video frame interpolation scenarios, we collected the Clear-Motion Test Sequences solely for testing purposes. 10.1. Event-RGB Aligned Vid...

  56. [64]

    The Impact of Input Upsampling As discussed in the main paper, to mitigate the loss of ap- pearance and motion control accuracy caused by the conver- sion between downsampled latent space and pixel space in Latent Diffusion Models (LDM) [7], we employ test-time optimization in...

  57. [65]

    More Visual Results In this section, we provide additional visual results show- casing qualitative comparisons between our method and the baselines, as shown in Figures 14, 15, 16, 17, 18, and 19

  58. [66]

    The pre-trained video diffusion model we used is Stable Video Diffusion [ 7] for 14-frame image-to-video genera- tion

    Additional Implementation Details In this section, we provide additional implementation details. The pre-trained video diffusion model we used is Stable Video Diffusion [ 7] for 14-frame image-to-video genera- tion. We trained our model with an effective batch size of 64, usin...

  59. [67]

    Each method generated 1024 × 576 frames with run time averaged over 16 frames

    Model Run Time, Memory, and Parameter Comparison Table 6 reports testing results for all models run on a single NVIDIA RTX 4090 GPU. Each method generated 1024 × 576 frames with run time averaged over 16 frames. VDM based methods (Time-Reversal, DynamiCrafter, and Ours) are mo...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.