Pith. sign in

REVIEW 3 major objections 5 minor 36 references

CI-VID: A Coherent Interleaved Text-Video Dataset

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that coherent multi-scene video generation needs interleaved text-video data with transition labels, and that the 340,000-sample CI-VID dataset provides it.

desk verdict CI-VID is a genuinely new dataset with a transparent pipeline, but the experiments never isolate the interleaved structure from extra data, so the headline claim about T&V2V is not yet proven. read the letter →

arxiv 2507.01938 v1 pith:OVOHJGUD submitted 2025-07-02 cs.CV

Yiming Ju , Jijin Hu , Zhengxiong Luo , Haoge Deng , hanyu Zhao , Li Du , Chengwei Wu , Donglin Hao
show 2 more authors
Xinlong Wang Tengfei Pan
This is my paper
classification cs.CV
keywords text-and-video-to-videogenerationinterleavedtext-videodatasetcoherentvideosequencemulti-clipcaptioningtemporalcoherencebenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that public text-video datasets, built as isolated one-to-one clip-caption pairs, cannot teach models to generate coherent multi-scene videos. It introduces CI-VID, a dataset of over 340,000 samples in which each sample is a sequence of related video clips accompanied by two caption layers: one describing each clip and one describing the transitions between adjacent clips, covering what continues and what changes in content, background, camera angle, and movement. This interleaved structure is meant to support text-and-video-to-video generation, where the model uses both the preceding clip and a text instruction to produce the next clip. The authors report that a 0.6 billion parameter video model fine-tuned on CI-VID clearly outperforms the same model without fine-tuning on human, vision-language-model, and similarity-based evaluations of consistency, narrativity, and correctness. If the result holds, it suggests that explicit inter-clip relational supervision is the missing ingredient for story-driven video generation.

What carries the argument

The load-bearing object is the CI-VID sample: a sequence of two to ten content-related but visually varied video clips paired with one individual caption per clip and one joint caption per adjacent pair. The construction pipeline combines similarity-based segmentation, which uses embedding cosine similarity with thresholds of 0.6 and 0.8 plus distance constraints to find scene boundaries, with entity-based segmentation, which extracts a main entity and verifies that the same entity or person appears across most clips to guarantee content relevance. Captions are generated in two passes, with sequential-frame input producing detailed per-clip descriptions of content, camera angle, camera movement, and background, and joint-frame input producing transition captions for content, background, camera angle, and camera movement. During training, clips and captions are tokenized and arranged into an interleaved input sequence, with supervision applied to the visual tokens through a diffusion loss.

What would settle it

Fine-tune the same base model on a control dataset built from the same clips and captions with the sequence structure removed, such as shuffled clip order and no joint captions while matching data volume and caption length, then run the same human and automatic evaluations; if the control matches or beats the sequence-trained model, the coherence advantage claimed for CI-VID is not demonstrated.

Watch

Extended reading notes

Core claim

The central claim is that coherent multi-scene video generation is better learned from interleaved text-video sequences than from isolated text-video pairs, and that CI-VID supplies that supervision at scale. Each CI-VID sample is a sequence of content-relevant but visually varied clips built from a source video, paired with individual captions for every clip and joint captions that explicitly record content continuation, content change, background continuation and change, and camera angle and movement change between adjacent clips. Fine-tuning a small autoregressive video model on CI-VID improves cross-clip consistency, narrative flow, factual correctness, and similarity to ground-truth subsequent clips relative to the same model without fine-tuning. The authors conclude that CI-VID enables text-and-video-to-video generation and supports smooth, story-driven output with strong temporal coherence.

Load-bearing premise

The load-bearing premise is that the measured improvements come from CI-VID's coherent multi-clip sequence structure, not from the larger volume of video-caption data or the unusually long captions alone.

Editorial extensions

If this is right

  • A video generation model can be trained to generate the next clip conditioned on the previous clip and a text instruction, instead of generating each scene from text alone.
  • Fine-tuning a text-to-video model on CI-VID yields large human-judged gains in cross-scene consistency, narrative flow, and factual correctness compared with the same model without fine-tuning.
  • VLM-based evaluation shows clear improvement on stylistic, entity, and background consistency as well as perspective transitions, while text-prompt alignment stays at least on par.
  • Similarity-based evaluation shows the fine-tuned model matches ground-truth subsequent clips better on holistic and object-level metrics.
  • Because more than 100,000 CI-VID samples contain four or more clips, the dataset supports training on long input sequences, not just pairwise adjacent-clip tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the CI-VID advantage transfers to larger models, the same interleaved supervision could push long-form storytelling by letting a model repeatedly consume the last generated clip and a scene description.
  • The joint captions explicitly annotate what changes and what continues between clips, so they could be reused for video editing tasks that require localizing which aspects of a scene to preserve and which to alter.
  • A natural next test is a controlled ablation that keeps data volume and caption length equal but removes sequence order and joint captions; that would isolate whether inter-clip structure or simply more and better captions drives the reported gains.
  • Because the pipeline already labels camera angle and movement changes, CI-VID could also support learning shot-transition predictors or camera-control modules for storyboard generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CI-VID, a dataset of 341,550 interleaved text-video sequences, each containing multiple video clips with individual captions and joint captions that describe continuity and change between adjacent clips. The construction pipeline starts from 592,429 YouTube videos, applies scene detection, motion and text filtering, then builds clip sequences using ImageBind similarity thresholds and an entity-based segmentation step that uses Qwen2.5-VL and GPT-4o to verify shared entities. Captions are generated with GPT-4o in a structured format covering content, camera, and background. The authors fine-tune a 0.6B NOVA video model on CI-VID and evaluate it against the unfine-tuned base using human evaluation, VLM scoring, and similarity metrics, reporting substantial gains on consistency, narrativity, and correctness. The central claim is that the interleaved coherent-sequence structure, not just additional video-text data, enables text-and-video-to-video generation.

Significance. If the central claim is substantiated, CI-VID would be a valuable and novel resource: it is, to my knowledge, the first large-scale public video dataset that explicitly provides inter-clip relational supervision through joint captions, and the authors release the dataset and construction/evaluation code. The pipeline is transparent, the dataset statistics are concrete, and the human evaluation reports high inter-annotator agreement. The similarity-based evaluation uses ground-truth videos and standard metrics, which is a positive independent check. However, the paper's headline empirical result is not yet established: the experiments compare only a CI-VID-fine-tuned model with the unfine-tuned base, conflating the interleaved structure with extra data volume and richer captions. This is a fixable but load-bearing gap, so the contribution is better described as a dataset resource with promising preliminary evidence rather than a validated demonstration of a new training paradigm.

major comments (3)
  1. [Sec. 4.2, Tables 4-6] The central causal claim that CI-VID's interleaved sequence structure improves coherent video generation is not isolated by the current experimental design. Every quantitative comparison in Sec. 4.2 contrasts the model fine-tuned on CI-VID with the same NOVA-0.6B model without any CI-VID fine-tuning. This contrast bundles at least three variables: (1) 341,550 additional training samples, (2) long structured captions of over 200 words on average, and (3) the joint captions and clip order that are the dataset's claimed novelty. An ablation is needed that trains the same model on the same clips and individual captions with the sequence structure and joint captions removed (or with clips shuffled), so the incremental contribution of the interleaved relational supervision can be measured. Without such a control, the reported win rates, including the 90.0% consistency figure in Table 4, cannot be attributed to the coherent-sequence structure.
  2. [Sec. 4.2.2, Table 5; Sec. 3.2.2] The VLM-based evaluation uses Qwen2.5-VL-72B-Instruct, the same model family used in dataset construction for main-entity extraction and filtering in Sec. 3.2.2. This introduces a risk of circularity for the entity-consistency and background-consistency dimensions of Table 5, since the evaluated model was trained on data that Qwen2.5-VL itself helped select. I am not claiming this is a deliberate bias, but the overlap is a real correctness risk. The authors should either use a different VLM family for evaluation, or report a correlation between the VLM scores and the independent human judgments on the same samples, per dimension, to show that the VLM axis is not merely reinforcing the construction filter.
  3. [Sec. 4.1 vs Sec. 4.2.1] The experimental setup contains an ambiguity about the baseline model. Sec. 4.1 says the model weights are initialized from the T2V model NOVA-0.6B, while Sec. 4.2.1 describes 'a video generation model trained on the large-scale Emu3 dataset' as the baseline that is then fine-tuned on CI-VID. The reader cannot tell whether the comparison in Tables 4-6 is between CI-VID-fine-tuned NOVA and unfine-tuned NOVA, or between CI-VID-fine-tuned Emu3-trained NOVA and unfine-tuned Emu3-trained NOVA. This should be clarified explicitly, along with the exact training data and steps used for the baseline, because it affects the interpretation of all reported gains.
minor comments (5)
  1. [Abstract] The abstract ends with 'underscoring the quality and practical utility of the CI-VID dataset' and is missing a closing period.
  2. [Figure 1] The figure contains the typo 'JOINT CAPTIION' in the caption label and the word 'romoving' in the example joint caption; these should be corrected.
  3. [Table 2 and Sec. 3.4] The text refers to 'MIRA' while the table lists 'MiraData'; please use the consistent name throughout.
  4. [Sec. 3.2.2] The bullet items under 'Entity-Based Segmentation' start with lowercase letters after the colon; capitalize 'Main entity extraction' and 'Clip entity examination' for consistency.
  5. [Sec. 3.4] The sentence 'Over 98% of the video clips have a resolution of 1080p or higher' is a strong claim; please specify how resolution was measured (e.g., from container metadata or decoded frames) and note whether the reported percentage covers all 1M clips.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: CI-VID's central claim rests on an independent fine-tuning comparison; the Qwen overlap is an evaluation-validity caveat, not a definitional reduction.

full rationale

The paper's derivation chain is empirical: collect videos, segment into coherent clip sequences using similarity and entity-based criteria, generate captions with GPT-4o, fine-tune NOVA-0.6B on the resulting interleaved samples, and compare against the unfine-tuned base. No reported number is constructed from its own target. The entity-based segmentation does use Qwen2.5-VL-72B-Instruct to filter sequences, and the VLM evaluation also uses Qwen2.5-VL-72B-Instruct for the entity-consistency axis, so the same model family is involved on both sides; however, the evaluation compares newly generated videos from two models, not the filtered training clips, and the human evaluation plus CLIP/LPIPS/SSIM metrics are independent of Qwen. The improvements could in principle be confounded by extra data volume, caption richness, or base-model initialization, but this is an experimental-control limitation rather than a circular reduction: no equation defines the outcome in terms of the input, and no fitted parameter is renamed as a prediction. Citations to Emu3 and NOVA are self-citations used as a data source and base model, respectively, but they are not invoked as proof of CI-VID's effectiveness. Thus no circular step meets the evidentiary bar in the instructions.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The dataset construction depends on a long list of hand-set thresholds and on the reliability of proprietary VLMs for entity and caption quality. These are assumptions about curation quality, not physical theory. The experimental claim rests additionally on evaluation choices made in Sec. 4 that are not fully controlled.

free parameters (6)
  • Similarity thresholds = Tl=0.6, Th=0.8
    Hand-set in Sec. 3.2.1 to balance segmentation quality against sample quantity.
  • Motion filtering threshold = 70 mean optical flow magnitude per pixel normalized by shorter edge
    Chosen in Sec. 3.1 to filter clips with insufficient motion strength.
  • Text coverage threshold = 10%
    Clips with over 10% on-screen text are discarded in Sec. 3.1.
  • Scene detection threshold = 3
    PySceneDetect threshold in Sec. 3.1, chosen to avoid shot transitions within clips.
  • Entity pass ratio = 70%
    Sequences are discarded if fewer than 70% of clips contain the main entity (Sec. 3.2.2).
  • Distance constraints = index difference <=3, time gap <=10s
    Used in Sec. 3.2.1 to preserve scene continuity after filtering.
assumptions (5)
  • domain assumption Cosine similarity of ImageBind embeddings of three concatenated frames reliably indicates scene-level semantic continuity and variation.
    Used as the core signal in Similarity-Based Segmentation (Sec. 3.2.1) to decide where sequences start, stop, and skip clips; thresholds are empirical.
  • domain assumption Qwen2.5-VL-72B correctly identifies the main entity and verifies the same person across clips.
    Entity-Based Segmentation (Sec. 3.2.2) discards sequences based on this VLM's answers, so dataset coherence inherits its accuracy.
  • domain assumption GPT-4o cross-validation removes errors left by Qwen2.5-VL.
    The revalidation step in Sec. 3.2.2 assumes a second VLM improves sequence quality; no error analysis supports this assumption.
  • domain assumption Human, VLM, and similarity metrics measure the intended T&V2V capabilities.
    Section 4.2 treats these metrics as evidence for accuracy and content consistency without a controlled baseline or statistical validation.
  • domain assumption Interleaved-data benefits shown for image-text models transfer to video generation.
    Section 2.2 generalizes findings from Flamingo, KOSMOS-1, and CoMM to the video setting without direct prior evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CI-VID: A Coherent Interleaved Text-Video Dataset." pith.science (2026). https://pith.science/paper/OVOHJGUD

@misc{pith2026250701938,
  author       = {Pith},
  title        = {Pith review of: CI-VID: A Coherent Interleaved Text-Video Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OVOHJGUD}},
  note         = {Machine review of arXiv:2507.01938}
}
read the original abstract

Text-to-video (T2V) generation has recently attracted considerable attention, resulting in the development of numerous high-quality datasets that have propelled progress in this area. However, existing public datasets are primarily composed of isolated text-video (T-V) pairs and thus fail to support the modeling of coherent multi-clip video sequences. To address this limitation, we introduce CI-VID, a dataset that moves beyond isolated text-to-video (T2V) generation toward text-and-video-to-video (TV2V) generation, enabling models to produce coherent, multi-scene video sequences. CI-VID contains over 340,000 samples, each featuring a coherent sequence of video clips with text captions that capture both the individual content of each clip and the transitions between them, enabling visually and textually grounded generation. To further validate the effectiveness of CI-VID, we design a comprehensive, multi-dimensional benchmark incorporating human evaluation, VLM-based assessment, and similarity-based metrics. Experimental results demonstrate that models trained on CI-VID exhibit significant improvements in both accuracy and content consistency when generating video sequences. This facilitates the creation of story-driven content with smooth visual transitions and strong temporal coherence, underscoring the quality and practical utility of the CI-VID dataset We release the CI-VID dataset and the accompanying code for data construction and evaluation at: https://github.com/ymju-BAAI/CI-VID

Figures

Figures reproduced from arXiv: 2507.01938 by the authors.

Figure 1
Figure 1. An example from the CI-VID dataset. Each sample consists of a sequence of video clips, individual captions describing each [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The pipeline for constructing CI-VID samples. The Segmentation modules construct clip sequences from the source video, while [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Word cloud of CI-VID sample themes, derived from [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (6 more)
Figure 3
Figure 3. Figure 3: The analysis of sample characteristics: (a) the number [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]
Figure 5
Figure 5. Figure 5: Comparison between generated results with and without finetuning on CI-VID. Sample (a) shows the generated result from the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Example of similarity-based evaluation setup. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: CI-VID Examples (1/6): Video sequences are extracted [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 11
Figure 11. Figure 11: CI-VID Examples (5/6): Video sequences are extracted [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: CI-VID Examples (6/6): Video sequences are extracted [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 19 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. ArXiv preprint, abs/2502.13923, 2025. 5, 8

    arXiv 2025

  3. [3]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1728–1738,

    work page

  4. [4]

    Video generation models as world simulators, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, et al. Video generation models as world simulators, 2024. 1, 2

    work page 2024

  5. [5]

    Muse: Text-to-image generation via masked generative transform- ers

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Mur- phy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transform- ers. ArXiv preprint, abs/2301.00704, 2023. 7

    arXiv 2023

  6. [6]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370–387. Springer, 2024. 1, 3, 4

    work page 2024

  7. [7]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13320–13331, 2024. 3, 4

    work page 2024

  8. [8]

    Comm: A coherent inter- leaved image-text dataset for multimodal understanding and generation

    Wei Chen, Lin Li, Yongqi Yang, Bin Wen, Fan Yang, Tingt- ing Gao, Yu Wu, and Long Chen. Comm: A coherent inter- leaved image-text dataset for multimodal understanding and generation. ArXiv preprint, abs/2406.10462, 2024. 3

    arXiv 2024

Show all 36 references
  1. [9]

    Yolo-world: Real-time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xing- gang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024. 8

    2024

  2. [10]

    Autoregressive video generation without vector quantization

    Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. ArXiv preprint, abs/2412.14169, 2024. 6, 7

    2024 arXiv

  3. [11]

    Long video generation with time-agnostic vqgan and time- sensitive transformer

    Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time- sensitive transformer. In European Conference on Computer Vision, pages 102–118. Springer, 2022. 1

    2022

  4. [12]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 15180–15190, 2023. 4

    2023

  5. [13]

    Language is not all you need: Aligning perception with language mod- els

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language mod- els. Advances in Neural Information Processing Systems , 36:72096–721...

    2023

  6. [14]

    Vbench: Comprehensive bench- mark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive bench- mark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  7. [15]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. ArXiv preprint, abs/2410.21276, 2024. 5

    2024 arXiv

  8. [16]

    Phi-2: The surprising power of small language models

    Mojan Javaheripi, S ´ebastien Bubeck, Marah Abdin, Jy- oti Aneja, Sebastien Bubeck, Caio C ´esar Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3):3, 2023. 7

    2023

  9. [17]

    Miradata: A large-scale video dataset with long durations and structured captions

    Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xin- tao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. Advances in Neural Information Processing Systems, 37:48955–48970, 2025. 1, 3, 4

    2025

  10. [18]

    Videopoet: A large language model for zero-shot video gen- eration

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vigh- nesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video gen- eration. ArXiv preprint, abs/2312.14125, 2023. 1, 2

    2023 arXiv

  11. [19]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents

    Hugo Laurenc ¸on, Lucile Saulnier, L´eo Tronchon, Stas Bek- man, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Sid- dharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Informa...

    2023

  12. [20]

    Autoregressive image generation without vec- tor quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization. Advances in Neural Information Processing Systems, 37:56424–56445, 2025. 6, 7

    2025

  13. [21]

    Open-sora plan: Open-source large video generation model

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Li- uhan Chen, et al. Open-sora plan: Open-source large video generation model. ArXiv preprint, abs/2412.00131, 2024. 6

    2024 arXiv

  14. [22]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. 7

    2019

  15. [23]

    Openvid-1m: A large-scale high-quality dataset for text-to- video generation

    Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhen- heng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to- video generation. ArXiv preprint, abs/2407.02371, 2024. 1, 3

    2024 arXiv

  16. [24]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR,

  17. [25]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  18. [26]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 3

    2020

  19. [27]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural in- f...

    2022

  20. [28]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  21. [29]

    Videotetris: Towards compositional text-to-video generation

    Ye Tian, Ling Yang, Haotian Yang, Yuan Gao, Yufan Deng, Jingmin Chen, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, et al. Videotetris: Towards compositional text-to-video generation. ArXiv preprint, abs/2406.04277, 2024. 1

    2024 arXiv

  22. [30]

    Mcvd-masked conditional video diffusion for prediction, generation, and interpolation

    Vikram V oleti, Alexia Jolicoeur-Martineau, and Chris Pal. Mcvd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in neural informa- tion processing systems, 35:23371–23385, 2022. 1

    2022

  23. [31]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. ArXiv preprint, abs/2409.18869, 2024. 1, 4

    2024 arXiv

  24. [32]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. ArXiv preprint, abs/2307.06942, 2023. 1, 3, 4

    2023 arXiv

  25. [33]

    Ad- vancing high-resolution video-language representation with large-scale video transcriptions

    Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Ad- vancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2022

  26. [34]

    Vript: A video is worth thousands of words

    Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. ArXiv preprint, abs/2406.06040, 2024. 1, 3, 4

    2024 arXiv

  27. [35]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. ArXiv preprint, abs/2408.06072, 2024. 1

    2024 arXiv

  28. [36]

    Multimodal c4: An open, billion-scale corpus of images interleaved with text

    Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.