REVIEW 6 cited by
M4V: Multimodal Mamba for Efficient Text-to-Video Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
M4V: Multimodal Mamba for Efficient Text-to-Video Generation
read the original abstract
Text-to-video generation has significantly enriched content creation and holds the potential to evolve into powerful world simulators. However, modeling the vast spatiotemporal space remains computationally demanding, particularly when employing Transformers, which incur quadratic complexity in sequence processing and thus limit practical applications. Recent advancements in linear-time sequence modeling, particularly the Mamba architecture, offer a more efficient alternative. Nevertheless, its plain design limits its direct applicability to multimodal and spatiotemporal video generation tasks. To address these challenges, we introduce M4V, a multimodal Mamba framework for efficient text-to-video generation. Specifically, a MultiModal diffusion Mamba (MM-DiM) block is designed within the framework to enable seamless integration of multimodal information and spatiotemporal modeling. In detail, we introduce a novel multimodal token re-composition design, which employs a bidirectional scheme for multimodal information integration through simple token arrangement, along with visual registers to enhance spatialtemporal consistency. As a result, the MM-DiM blocks in M4V reduce FLOPs by 45% compared with the attention-based alternative when generating videos at 768x1280 resolution. Additionally, several training strategies are explored in this work to provide a better understanding of training text-to-video models using only publicly available datasets. Extensive experiments on text-to-video benchmarks demonstrate M4V's ability to produce high-quality videos while significantly lowering computational costs. Project page: https://huangjch526.github.io/M4V_project/.
Forward citations
Cited by 6 Pith papers
-
Setting the Stage: Text-Driven Scene-Consistent Image Generation
A new data pipeline using real photos, entity removal, and image-to-video models plus a cross-view attention loss enables text-driven generation of actors in reference scenes with improved alignment.
-
MobileWan: Closing the Quality Gap for Mobile Video Diffusion
A 5B-parameter video diffusion transformer is made deployable on mobile hardware via recurrence distillation, learnable head pruning, step distillation, and decoder optimization, achieving 83.79 VBench at 20s latency.
-
OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation
OmniShow unifies text, image, audio, and pose conditions into an end-to-end model for high-quality human-object interaction video generation and introduces the HOIVG-Bench benchmark, claiming state-of-the-art results.
-
SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing
Parameter-free centroid compensation plus error-aware block routing yields a better quality–density Pareto frontier for sparse attention in video DiTs than score-based sparsification.
-
FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
FSDrive uses a generated future scene frame as visual spatio-temporal CoT to improve VLA models for safer autonomous driving trajectory prediction.
-
MobileWan: Closing the Quality Gap for Mobile Video Diffusion
A 5B-parameter video diffusion transformer can be compressed and reformulated as an RNN to generate 5s 480x832 videos on a Snapdragon 8 Gen 5 mobile NPU in about 20 seconds with a VBench score of 83.79.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.