Pith. sign in

REVIEW 6 major objections 4 minor 55 references

BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations

T0 review · 6 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that grounding video diffusion models in moving tilted ellipses paired with per-object captions—blob video representations—delivers stronger layout control and cross-frame object consistency than bounding-box layouts…

desk verdict A solid, incremental extension of BlobGEN to video with a real evaluation-bias caveat; deserves a serious referee. read the letter →

arxiv 2501.07647 v1 pith:ZDGD7KZ4 submitted 2025-01-13 cs.CV cs.AI

Weixi Feng , Chao Liu , Sifei Liu , William Yang Wang , Arash Vahdat , Weili Nie This is my paper · ORCID
classification cs.CVcs.AI
keywords blobvideorepresentationlayout-guidedgenerationtext-to-videocompositionaldiffusionmodelsmasked3Dattentionobject-leveltemporalconsistencyLLMlayoutplanning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a video can be decomposed into blob video representations—each object encoded as a moving tilted ellipse plus a free-form caption on selected frames—and that these blobs are a better grounding input for text-to-video diffusion than bounding boxes. The central claim is that a blob-grounded diffuser, with masked attention that ties each object to the same pixels across frames, simultaneously improves layout adherence, object-level consistency, and prompt alignment. The paper builds a one-million-video annotated dataset, instantiates the framework on both U-Net and transformer video backbones, and reports that on layout-to-video benchmarks it raises mIOU by more than twenty percent over the strongest box-based baseline. It further claims that when a large language model plans the blobs, the full pipeline exceeds commercial text-to-video systems on dynamic attribute binding, spatial relations, motion binding, and numerical accuracy.

What carries the argument

The load-bearing object is the blob video representation: for each of N objects and each of T frames a vector $\tau = [c_x, c_y, a, b, \theta]$ defines a tilted ellipse, and every k-th frame pairs that ellipse with a free-form caption describing the object's appearance. The argument is carried by two attention modules: masked spatial cross-attention, which fuses each blob embedding only with visual features inside its own ellipse in the same frame, and masked 3D self-attention, which flattens frames and lets a feature in one frame attend only to the same object (or the same background) in all frames. A context-interpolation module fills caption embeddings for frames without descriptions by interpolating between neighboring anchor-frame embeddings in the text-embedding space. Because only the new layers are trained while the pre-trained video diffusion weights are frozen, the modules act as an adapter that preserves prior video quality while adding per-object control.

What would settle it

Re-run the layout-to-video comparison scoring object regions with human annotation, or with a video-instance-segmentation model trained on data independent of the annotation toolchain, and compare mIOU and rCLIPt between BlobGEN-Vid and the box-based baselines; if the relative advantage disappears or reverses under a different scorer, the reported layout-control gain is at least partly a metric artifact rather than a true property of blob grounding.

Watch

Extended reading notes

Core claim

BlobGEN-Vid introduces blob video representations as the condition for text-to-video generation: each object instance carries a five-parameter tilted ellipse in every frame and a free-form visual description on anchor frames spaced every k frames. The method's core discovery is that this representation, combined with two masked attention mechanisms, fixes two weaknesses of box-based layout grounding: boxes cannot express orientation or fine-grained appearance, and per-pixel temporal attention does not keep one object's pixels tied to the same object over time. The masked spatial cross-attention makes each blob attend only to its ellipse region in each frame, and the masked 3D self-attention makes each object's features across all frames attend to each other, with background attending only to background. A context-interpolation module linearly (or via a learned Perceiver-style network) interpolates caption embeddings between anchor frames, giving non-annotated frames smooth semantic context. Empirically the paper claims superior zero-shot video generation and state-of-the-art layout controllability on YoutubeVIS-2021, and, with LLM-planned blobs, better compositional accuracy than commercial generators on T2V-CompBench and TC-Bench.

Load-bearing premise

The reported layout-control and consistency scores are computed by detecting and tracking objects with the same family of models used to create the blob training annotations, so the measured advantage could be inflated if those models score blob-shaped outputs more favorably than box-based outputs.

Editorial extensions

If this is right

  • Blob layouts can be written as structured text, so an LLM can plan motion, positions, and counts from a prompt, and a human can edit a video by editing the ellipse tracks.
  • The same framework covers motion control, camera control, appearance transitions, and numerical accuracy, because orientation, per-object captions, and temporal interpolation are all first-class parameters.
  • Masked 3D attention should make object identity persist through occlusion and large motion more reliably than models with only per-pixel temporal self-attention.
  • The adapter-style training means the blob grounding can be carried onto new video diffusion backbones as they appear, without retraining the base model.
  • A blob planning stage removes the need for manual layout annotation in many text-to-video applications, since the LLM produces the JSON layout before generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: a natural extension is treating blob tracks as an editable keyframe animation layer—users could move, rotate, resize, or swap ellipses over time and regenerate the video, effectively editing motion by editing a few ellipses.
  • Beyond the paper: a testable disentanglement check follows from the representation's design—identical blob tracks with swapped blob captions should change object appearance while preserving geometry; if captions leak into shape or position, the two components are not as separable as claimed.
  • Beyond the paper: the one-million-video annotation pipeline suggests a self-training loop in which generated videos are re-segmented into blobs and used to train the next generation of the model, reducing reliance on external annotation.
  • Beyond the paper: long-video generation could be achieved by running the LLM planner autoregressively over chunks and connecting blob tracks across chunk boundaries; the paper's sparse-anchor-frame design already anticipates arbitrary anchor placement through its Perceiver-based interpolation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 4 minor

Summary. The paper proposes BlobGEN-Vid, a layout-guided text-to-video generation framework that conditions video diffusion models on blob video representations, which consist of per-frame tilted ellipses (blob parameters) paired with free-form text descriptions. The method introduces a masked 3D self-attention module to enforce object-level temporal consistency, a context interpolation module to handle sparse blob captions across frames, and an LLM-based pipeline for automatic blob layout generation. The framework is implemented on both U-Net (VideoCrafter2) and DiT (CogVideoX-5B) backbones. Experiments on Youtube-VIS 2021, T2V-CompBench, TC-Bench, and ScanNet++ report improvements over layout-guided baselines (TrackDiffusion, LVD, VideoTetris) and proprietary text-to-video systems, with ablations supporting the proposed architectural choices.

Significance. If the results are robust, blob video representations offer a more expressive yet still lightweight conditioning layout than bounding boxes for controllable video generation. The masked 3D attention mechanism is a simple and plausible way to improve object-level consistency, and the demonstration of the framework on two backbone architectures shows model-agnostic applicability. The paper also provides extensive ablations on context interpolation and data scaling. However, the evaluation has several load-bearing weaknesses, notably a potential distributional bias from using the same detector-tracker for both annotation and evaluation, non-uniform frame protocols across baselines, and unsupported claims such as 'zero-shot'. These issues must be addressed before the state-of-the-art claims are convincing. The strengths are the clear method presentation, the detailed appendix, and the systematic ablations.

major comments (6)
  1. [Sec. 5.1 and Appendix C (Evaluation metrics)] The layout controllability metrics mIOU, rCLIPt, and rCFC are computed by applying Grounding DINO + SAM2 to generated videos, which is the same detector-tracker family used to create the training blob annotations. This creates a risk of distributional bias: BlobGEN-Vid's outputs may be more easily detected and tracked by that toolchain than the outputs of baselines, inflating its scores even if the generated layouts are equally valid. The authors should provide control evidence, such as an independent detector/tracker, a human evaluation on a subset, or per-method detection success rates, to rule out this bias.
  2. [Appendix C (Evaluation metrics for Table 1)] The evaluation protocols in Table 1 differ per baseline: TrackDiffusion is scored on all 16 frames, LVD on frames 1, 4, 7, 10, 13, 16, VideoTetris on frames 9, 17, 25 of 32, BlobGEN-Vid (VC2) on all 16 frames, and BlobGEN-Vid (CogVideoX) on every 4th frame of 49. Because mIOU, rCLIPt, and rCLIPi are measured only on the frames with layouts, the methods are compared under different temporal samplings and video lengths. The reported margins may partly reflect this protocol mismatch; the authors should evaluate all methods on the same frame indices and video length, or justify why the differences do not affect the comparison.
  3. [Sec. 4.2, Eq. (2)] The context interpolation formula has the weighting coefficients reversed. For a non-anchor frame t between tk and tk+1, the equation assigns weight (tk+1 - t)/k to e_s(tk+1) and weight (t - tk)/k to e_s(tk). Thus near t = tk the interpolated embedding is close to the right anchor's embedding, and near t = tk+1 it is close to the left anchor's embedding, which is the opposite of a linear interpolation. The correct convex combination should be (t - tk)/k * e_s(tk+1) + (tk+1 - t)/k * e_s(tk). Since the experiments reportedly demonstrate improvements from context interpolation, this is likely a typographical error in the paper, but it must be corrected and the implementation made explicit.
  4. [Table 3 and Sec. 5.4] Adding the masked 3D attention module (row 7 vs row 8) improves PSNR, CFC, and rCFC but worsens FVD from 142 to 161. The paper's claim that masked 3D attention 'facilitates video diffusion models to generate consistent objects' is supported by the consistency metrics, but the opposite movement of FVD is not discussed beyond a general caveat that FVD is not robust. Because the introduction and Sec. 5.4 also imply quality improvements, the authors should explain why FVD is not meaningful here or report an additional quality metric that corroborates the consistency gains.
  5. [Abstract and Sec. 1] The phrase 'zero-shot video generation ability' is undefined and unsupported. All experiments fine-tune BlobGEN-Vid on large annotated video datasets (Sec. 5.1), so 'zero-shot' cannot refer to no task-specific fine-tuning; if it refers to the ability to use blobs extracted from unseen videos or generated by an LLM without per-video adaptation, this is a different meaning that is not formalized or directly tested. The authors should either define the term precisely and provide an experiment that demonstrates it, or remove it from the central claims.
  6. [Table 1 and Sec. 2] Boximator, a closely related layout-guided video diffusion method, is discussed in the related work but is not included in the quantitative comparison. Without a direct evaluation against Boximator, the claim of state-of-the-art layout controllability is incomplete, especially because Boximator targets bounding-box conditioning and motion control similar to the proposed method.
minor comments (4)
  1. [Sec. 2] The word 'controllabbility' in the last paragraph on layout-guided video generation is a typo and should be 'controllability'.
  2. [Fig. 3 caption] The attention mask description says 'High-value elements in the 3D attention mask in the figure will be mapped to 0 while low-value elements are mapped to −∞ as in Eq. 4'; this wording is confusing because Eq. 4 maps same-object and background pairs to 0 and all other pairs to −∞. A simplified illustration of the mask or a clearer caption would help.
  3. [Tables 1 and 4] No error bars, confidence intervals, or significance tests are reported, and several differences are small (e.g., rCLIPt 0.2794 vs 0.2763 for linear vs slerp in Table 4). Multiple runs or bootstrap confidence intervals would strengthen the reliability of the comparisons.
  4. [Sec. 4.3 and Table 6] The paper does not report the success rate of GPT-4o in producing parseable JSON layouts or the frequency of layout failures, which would help assess the robustness of the LLM-based pipeline beyond the aggregate benchmark scores.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core contributions are architectural improvements and empirical comparisons against external benchmarks and baselines; self-citations to BlobGEN and TC-Bench are contextual, not load-bearing in the derivation chain.

full rationale

The paper's derivation chain is self-contained in the sense required by the circularity check. The blob video representation is defined directly (Section 3 and 4.1) as tilted ellipses plus free-form captions, and the method modules—masked spatial cross-attention, masked 3D self-attention, and context interpolation—are explicitly defined by Eqs. (1)-(4) and then ablated in Tables 3-5. No equation or architectural component reduces the reported predictions to the training annotations or to a fitted parameter. The evaluation uses external benchmarks (YouTubeVIS-2021, T2V-CompBench, TC-Bench, ScanNet++) and external baselines (TrackDiffusion, LVD, VideoTetris, proprietary generators), so the central superiority claims are not forced by construction. The self-citations to BlobGEN and BlobGEN-3D are prior work that motivates the representation, but the video-specific extension and its evaluation are independent. TC-Bench is authored by the first author, but it is a fixed, externally defined benchmark with pre-specified metrics; using it does not make the result circular. A separate validity concern is that the YouTubeVIS metrics use Grounding DINO+SAM2 for detection and tracking, the same tool family used to create blob annotations, which could bias automated scores; however, this is a measurement-confounding risk, not a circular derivation, and per the hard rules it does not raise the circularity score. Overall, no step in the paper's claimed derivation is equivalent to its own inputs by definition.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method introduces a new representation (blob video) but no new physical entities. The central claim rests on domain assumptions about sufficient conditioning, frozen-backbone fine-tuning, CLIP embedding interpolation, and detection-based metrics. The listed free parameters are design choices that were not systematically varied or released.

free parameters (5)
  • Anchor frame interval k = 8
    Blob captions are annotated every k frames (k=8) for open-domain data; this design choice affects context interpolation quality and is not ablated in the paper.
  • Mask coverage threshold = 20%
    In the annotation pipeline, if LLaVA-NeXT+Grounding DINO mask coverage is below 20% of frame size, ODISE is used instead; this threshold controls the composition of the training data.
  • ScanNet++ sampling stride = 8
    16-frame clips are sampled from 128-frame ScanNet++ videos with stride 8, creating overlapping sub-clips, affecting multi-view training and evaluation.
  • Training data mix = 160K OpenVid + 460K VidGEN + 320K HDVILA
    The heavy filtering and balance between human and non-human videos are hand-chosen; the resulting mix is a free input to the method.
  • In-context exemplars for GPT-4o = Two fixed exemplars
    The LLM layout planning uses two hand-crafted exemplar prompts; performance depends on this prompt design, which is not varied.
assumptions (5)
  • domain assumption Pre-trained video diffusion models can be adapted by freezing their weights and training only newly inserted attention layers without degrading generation quality.
    The method relies entirely on fine-tuning new masked attention layers on top of frozen VC2 and CogVideoX backbones; if freezing prevents adaptation, the central results would not hold. Invoked in Sec 4.2.
  • domain assumption Blob ellipses (5 parameters) plus free-form captions are a sufficient grounding signal to control object location, motion, appearance, and camera motion.
    This is the core representational claim of the paper; it is validated only indirectly through the benchmark numbers. Stated in Sec 4.1.
  • domain assumption Linear interpolation of CLIP text embeddings between anchor frames yields smooth and semantically valid intermediate object descriptions.
    Context interpolation assumes continuity in CLIP embedding space; the paper's ablations show linear interpolation beats slerp but does not test against ground-truth intermediate captions. See Eq. 2 in Sec 4.2.
  • domain assumption SAM2 tracking and ellipse fitting produce blob sequences that accurately represent object motion in the training videos.
    The entire training set is built from automatic tracking and ellipse fitting; tracking errors would inject noisy conditions into training. Described in Sec 5.1 and Appendix A.
  • domain assumption The evaluation metrics mIOU, rCLIPt, rCFC, computed with detection and tracking models, faithfully measure layout controllability and object consistency.
    These metrics are central to the state-of-the-art claim and depend on the same Grounding DINO plus SAM2 toolchain used for annotation. See Sec 5.1 and Appendix C.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations." pith.science (2026). https://pith.science/paper/ZDGD7KZ4

@misc{pith2026250107647,
  author       = {Pith},
  title        = {Pith review of: BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZDGD7KZ4}},
  note         = {Machine review of arXiv:2501.07647}
}
read the original abstract

Existing video generation models struggle to follow complex text prompts and synthesize multiple objects, raising the need for additional grounding input for improved controllability. In this work, we propose to decompose videos into visual primitives - blob video representation, a general representation for controllable video generation. Based on blob conditions, we develop a blob-grounded video diffusion model named BlobGEN-Vid that allows users to control object motions and fine-grained object appearance. In particular, we introduce a masked 3D attention module that effectively improves regional consistency across frames. In addition, we introduce a learnable module to interpolate text embeddings so that users can control semantics in specific frames and obtain smooth object transitions. We show that our framework is model-agnostic and build BlobGEN-Vid based on both U-Net and DiT-based video diffusion models. Extensive experimental results show that BlobGEN-Vid achieves superior zero-shot video generation ability and state-of-the-art layout controllability on multiple benchmarks. When combined with an LLM for layout planning, our framework even outperforms proprietary text-to-video generators in terms of compositional accuracy.

Figures

Figures reproduced from arXiv: 2501.07647 by the authors.

Figure 1
Figure 1. With blob video representations, BlobGEN-Vid can support fine-grained controllability in text-to-video generation in terms of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Blob video representations for video generation consist [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. BlobGEN-Vid architecture with U-Net backbone or DiT backbone. Our method leverages two masked attention modules that [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Layout-to-video generation results on YoutubeVIS-2021 [ [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results on ScanNet++, where our method, es [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Our data annotation pipeline for obtaining blob video [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: An illustration of the context interpolation stage using a Perceiver-based model. Note that we omit the batch size ( [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Attention maps between a visual token and other visual tokens of the first frame. Some of the maps show similar spatial structure as Frame 1 in the pixel space. The visualization proves that full 3D attention still preserves the spatial structure as in UNet-based diffu…
Figure 9
Figure 9. Figure 9: Attention maps between a text token and other visual tokens of the first frame. The highlighted regions represent the regions where the text token is highly correlated with. The visualization proves that full 3D attention still preserves the spatial structure as in UNe…
Figure 10
Figure 10. Figure 10: An example of our data annotation results using instance list and Grounding DINO for segmentation. The text color of the blob [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: An example of our data annotation results using ODISE as the panoptic segmentation model. ODISE tends to segment the [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Qualitative examples from YoutubeVIS-700 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Qualitative examples from YoutubeVIS-700 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Qualitative examples from T2V-CompBench 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Qualitative examples from T2V-CompBench 21 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Qualitative examples from T2V-CompBench 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Qualitative examples from T2V-CompBench 23 [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Qualitative examples from TC-Bench 24 [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: Qualitative examples from TC-Bench 25 [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: Qualitative examples from ScanNet++ 26 [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 27 canonical work pages

  1. [1]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2, 5

    arXiv 2023

  2. [2]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 2, 3

    work page 2023

  3. [3]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 3

    arXiv

  4. [4]

    Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7310– 7320, 2024. 2, 3, 5, 13

    work page 2024

  5. [5]

    Training-free layout control with cross-attention guidance

    Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5343–5353, 2024. 2

    2024

  6. [6]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13320–13331, 2024. 2, 3

    work page 2024

  7. [7]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learn- ing, 2024. 3

    2024

  8. [8]

    Tc-bench: Benchmark- ing temporal compositionality in text-to-video and image-to- video generation

    Weixi Feng, Jiachen Li, Michael Saxon, Tsu-jui Fu, Wenhu Chen, and William Yang Wang. Tc-bench: Benchmark- ing temporal compositionality in text-to-video and image-to- video generation. arXiv preprint arXiv:2406.08656, 2024. 3, 6, 8, 15

    arXiv 2024

Show all 55 references
  1. [9]

    Layoutgpt: Compositional visual plan- ning and generation with large language models

    Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Ar- jun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual plan- ning and generation with large language models. Advances in Neural Information Processing Systems, 36, 2024. 3, 6

    2024

  2. [10]

    On the content bias in fr ´echet video distance

    Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun- Yan Zhu, and Jia-Bin Huang. On the content bias in fr ´echet video distance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7277– 7288, 2024. 8

    2024

  3. [11]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 13

    2022 arXiv

  4. [12]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2, 3

    2020

  5. [13]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In The Eleventh International Conference on Learning Representations, 2023. 2

    2023

  6. [14]

    Perceiver io: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Kop- pula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021. 5, 12

    2021 arXiv

  7. [15]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 4401–4410, 2019. 8

    2019

  8. [16]

    Open-sora-plan, 2024

    PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, 2024. 3

    2024

  9. [17]

    Dense optical tracking: Connecting the dots

    Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Dense optical tracking: Connecting the dots. InCVPR, 2024. 15

    2024

  10. [18]

    T2v- turbo: Breaking the quality bottleneck of video consis- tency model with mixed reward feedback

    Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sug- ato Basu, Wenhu Chen, and William Yang Wang. T2v- turbo: Breaking the quality bottleneck of video consis- tency model with mixed reward feedback. arXiv preprint arXiv:2405.18750, 2024. 2, 3

    2024 arXiv

  11. [19]

    Trackd- iffusion: Multi-object tracking data generation via diffusion models

    Pengxiang Li, Zhili Liu, Kai Chen, Lanqing Hong, Yunzhi Zhuge, Dit-Yan Yeung, Huchuan Lu, and Xu Jia. Trackd- iffusion: Multi-object tracking data generation via diffusion models. arXiv preprint arXiv:2312.00651, 2023. 2, 3, 4, 7, 15

    2023 arXiv

  12. [20]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023. 2, 3

    2023

  13. [21]

    Llm- grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models

    Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Llm- grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. Transactions on Machine Learning Research, 2024. 6

    2024

  14. [22]

    Llm-grounded video diffusion models

    Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li. Llm-grounded video diffusion models. In The Twelfth International Conference on Learning Representa- tions, 2024. 2, 3, 7, 8, 15

    2024

  15. [23]

    Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning

    Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning. arXiv preprint arXiv:2309.15091,

    arXiv

  16. [24]

    Motionclone: Training-free motion cloning for controllable video generation

    Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free motion cloning for controllable video generation. arXiv preprint arXiv:2406.05338 , 2024. 13

    2024 arXiv

  17. [25]

    9 Blobgen-3d: Compositional 3d-consistent freeview image generation with 3d blobs

    Chao Liu, Weili Nie, Sifei Liu, Abhishek Badki, Hang Su, Morteza Mardini, Benjamin Eckart, and Arash Vahdat. 9 Blobgen-3d: Compositional 3d-consistent freeview image generation with 3d blobs. In SIGGRAPH Asia 2024 Con- ference Papers, 2024. 3, 6, 7, 8

    2024

  18. [26]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 6, 12, 15

    2024

  19. [27]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 6, 12, 14, 15

    2023 arXiv

  20. [28]

    Evalcrafter: Benchmarking and eval- uating large video generation models

    Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and eval- uating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  21. [29]

    Fetv: A bench- mark for fine-grained evaluation of open-domain text-to- video generation

    Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou. Fetv: A bench- mark for fine-grained evaluation of open-domain text-to- video generation. Advances in Neural Information Process- ing Systems, 36, 2024. 3

    2024

  22. [30]

    Openvid-1m: A large-scale high-quality dataset for text-to- video generation

    Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhen- heng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to- video generation. arXiv preprint arXiv:2407.02371, 2024. 6, 13

    2024 arXiv

  23. [31]

    Compositional text-to-image gen- eration with dense blob representations

    Weili Nie, Sifei Liu, Morteza Mardani, Chao Liu, Benjamin Eckart, and Arash Vahdat. Compositional text-to-image gen- eration with dense blob representations. In Forty-first Inter- national Conference on Machine Learning, 2024. 2, 3, 7

    2024

  24. [32]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  25. [33]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 6, 12, 14

    2024 arXiv

  26. [34]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2

    2022

  27. [35]

    T2v-compbench: A comprehen- sive benchmark for compositional text-to-video generation

    Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2v-compbench: A comprehen- sive benchmark for compositional text-to-video generation. arXiv preprint arXiv:2407.14505, 2024. 3, 6, 8, 15

    2024 arXiv

  28. [36]

    Vidgen-1m: A large-scale dataset for text-to-video genera- tion

    Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, and Hao Li. Vidgen-1m: A large-scale dataset for text-to-video genera- tion. arXiv preprint arXiv:2408.02629, 2024. 6, 13

    2024 arXiv

  29. [37]

    Fourier features let networks learn high frequency functions in low dimen- sional domains

    Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ra- mamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimen- sional domains. Advances in neural information ...

    2020

  30. [38]

    Videotetris: Towards compositional text-to-video generation

    Ye Tian, Ling Yang, Haotian Yang, Yuan Gao, Yufan Deng, Jingmin Chen, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, et al. Videotetris: Towards compositional text-to-video generation. arXiv preprint arXiv:2406.04277, 2024. 3, 6, 7, 8, 15

    2024 arXiv

  31. [39]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6

    2018 arXiv

  32. [40]

    Score-based generative modeling in latent space

    Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in neural in- formation processing systems, 34:11287–11302, 2021. 3

    2021

  33. [41]

    Modelscope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023. 3

    2023 arXiv

  34. [42]

    Boximator: Gener- ating rich and controllable motions for video synthesis

    Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guo- qiang Wei, Liping Yuan, and Hang Li. Boximator: Gener- ating rich and controllable motions for video synthesis. In Forty-first International Conference on Machine Learning ,

  35. [43]

    Lavie: High-quality video gener- ation with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video gener- ation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023. 3

    2023 arXiv

  36. [44]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016. 2

    2016

  37. [45]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 2955–2966, 2023. 6, 12

    2023

  38. [46]

    Ad- vancing high-resolution video-language representation with large-scale video transcriptions

    Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Ad- vancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2022

  39. [47]

    The 3rd large-scale video object segmentation challenge - video in- stance segmentation track, 2021

    Linjie Yang, Yuchen Fan, Yang Fu, and Ning Xu. The 3rd large-scale video object segmentation challenge - video in- stance segmentation track, 2021. 6, 7

    2021

  40. [48]

    Compositional video generation as flow equalization

    Xingyi Yang and Xinchao Wang. Compositional video generation as flow equalization. arXiv preprint arXiv:2407.06182, 2024. 3

    2024 arXiv

  41. [49]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2, 3, 8, 13

    2024 arXiv

  42. [50]

    Scannet++: A high-fidelity dataset of 3d in- door scenes

    Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d in- door scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12–22, 2023. 6, 8 10

    2023

  43. [51]

    Show-1: Marrying pixel and latent diffusion models for text-to-video generation

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. International Journal of Com- puter Vision, pages 1–15, 2024. 3

    2024

  44. [52]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2, 7

    2023

  45. [53]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 12

    2024

  46. [54]

    both foreground and background

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 3, 8 11 BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations...

    2024

  47. [55]

    Frame0”: “Object2

    Therefore, the existence of interpolation for context fea- tures is essential to generate consistent videos and enhance prompt-video alignment. D.1. Additional qualitative results We show additional qualitative results from various settings and benchmarks in Fig. 12-20. 15 Gen...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.