MTCurv regresses pixel-wise microtubule curvature maps from noisy images using an attention-based residual U-Net trained on synthetic data with a gradient-aware loss.
Deep multi-scale video prediction beyond mean square error
11 Pith papers cite this work, alongside 1 external citations. Polarity classification is still indexing.
11
Pith papers citing it
1
external citations · Pith
abstract
Learning to predict future images from a video sequence involves the construction of an internal representation that models the image evolution accurately, and therefore, to some degree, its content and dynamics. This is why pixel-space video prediction may be viewed as a promising avenue for unsupervised feature learning. In addition, while optical flow has been a very studied problem in computer vision for a long time, future frame prediction is rarely approached. Still, many vision applications could benefit from the knowledge of the next frames of videos, that does not require the complexity of tracking every pixel trajectories. In this work, we train a convolutional network to generate future frames given an input sequence. To deal with the inherently blurry predictions obtained from the standard Mean Squared Error (MSE) loss function, we propose three different and complementary feature learning strategies: a multi-scale architecture, an adversarial training method, and an image gradient difference loss function. We compare our predictions to different published results based on recurrent neural networks on the UCF101 dataset
representative citing papers
A mixture-of-experts transformer foundation model pretrained on diverse SEM images enables generalization across materials and outperforms SOTA on unsupervised defocus-to-focus restoration.
Imagen Video generates high-definition text-conditional videos via a cascade of base and super-resolution diffusion models, achieving high fidelity and controllability.
Samudra 2 scales autoregressive neural ocean emulators to finer resolutions with architectural tweaks and dynamic loss, raising upper-ocean temperature R² from 0.56 to 0.87 at 1° and recovering mesoscale features.
MetaCLIP curates balanced 400M-pair subsets from CommonCrawl that outperform CLIP data, reaching 70.8% zero-shot ImageNet accuracy on ViT-B versus CLIP's 68.3%.
MagicVideo generates 256x256 text-conditioned video clips via latent diffusion with a custom 3D U-Net, achieving roughly 64 times lower compute than prior video diffusion models.
Single-level feature-to-feature forecasting with deformable convolutions on coarse abstract features from a segmentation backbone achieves state-of-the-art results for nine-timestep future semantic segmentation on Cityscapes validation.
SEE-Net improves video prediction by using frame shuffling to enforce learning of natural temporal order, reporting state-of-the-art results on three synthetic and real-world datasets.
DGF explicitly models multiple mode-conditioned predictive distributions via Dirichlet-guided sampling and reward optimization to preserve dynamical features in time series forecasts.
Trajectory-guided diffusion synthesis reconstructs missing frames in top-down drone videos of maritime maneuvers, outperforming optical flow extrapolation and RIFE interpolation on perceptual quality, motion realism, and trajectory adherence metrics.
PROM-Net performs unsupervised visual prediction of robot motion from raw frames and integrates the predictions into model predictive control for navigation in unknown dynamic settings.
citing papers explorer
-
MTCurv: Deep learning for direct microtubule curvature mapping in noisy fluorescence microscopy images
MTCurv regresses pixel-wise microtubule curvature maps from noisy images using an attention-based residual U-Net trained on synthetic data with a gradient-aware loss.
-
A Mixture of Experts Foundation Model for Scanning Electron Microscopy Image Analysis
A mixture-of-experts transformer foundation model pretrained on diverse SEM images enables generalization across materials and outperforms SOTA on unsupervised defocus-to-focus restoration.
-
Imagen Video: High Definition Video Generation with Diffusion Models
Imagen Video generates high-definition text-conditional videos via a cascade of base and super-resolution diffusion models, achieving high fidelity and controllability.
-
Samudra 2: Scaling Ocean Emulators across Resolutions
Samudra 2 scales autoregressive neural ocean emulators to finer resolutions with architectural tweaks and dynamic loss, raising upper-ocean temperature R² from 0.56 to 0.87 at 1° and recovering mesoscale features.
-
Demystifying CLIP Data
MetaCLIP curates balanced 400M-pair subsets from CommonCrawl that outperform CLIP data, reaching 70.8% zero-shot ImageNet accuracy on ViT-B versus CLIP's 68.3%.
-
MagicVideo: Efficient Video Generation With Latent Diffusion Models
MagicVideo generates 256x256 text-conditioned video clips via latent diffusion with a custom 3D U-Net, achieving roughly 64 times lower compute than prior video diffusion models.
-
Single Level Feature-to-Feature Forecasting with Deformable Convolutions
Single-level feature-to-feature forecasting with deformable convolutions on coarse abstract features from a segmentation backbone achieves state-of-the-art results for nine-timestep future semantic segmentation on Cityscapes validation.
-
Order Matters: Shuffling Sequence Generation for Video Prediction
SEE-Net improves video prediction by using frame shuffling to enforce learning of natural temporal order, reporting state-of-the-art results on three synthetic and real-world datasets.
-
Dirichlet-Guided Group Forecasting for Alleviating Over-smoothing in Time Series Forecasting
DGF explicitly models multiple mode-conditioned predictive distributions via Dirichlet-guided sampling and reward optimization to preserve dynamical features in time series forecasts.
-
Video Reconstruction using Diffusion-based Image-to-Video Generation with Trajectory Guidance
Trajectory-guided diffusion synthesis reconstructs missing frames in top-down drone videos of maritime maneuvers, outperforming optical flow extrapolation and RIFE interpolation on perceptual quality, motion realism, and trajectory adherence metrics.
-
Planning Robot Motion using Deep Visual Prediction
PROM-Net performs unsupervised visual prediction of robot motion from raw frames and integrates the predictions into model predictive control for navigation in unknown dynamic settings.