Pith. sign in

REVIEW 5 major objections 7 minor 60 references

InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption

T0 review · 5 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read InstanceCap claims that decomposing video captions into per-instance structured descriptions improves text-to-video fidelity and reduces hallucinations, supported by a 22K dataset and an inference-time prompt enhancer.

desk verdict InstanceCap has a plausible recipe and good reconstruction evidence, but its headline finetuning gain is confounded by an unmatched inference-time prompt enhancer. read the letter →

arxiv 2412.09283 v1 pith:GDU2EBDO submitted 2024-12-12 cs.CV cs.AI

Tiehan Fan , Kepan Nan , Rui Xie , Penghao Zhou , Zhenheng Yang , Chaoyou Fu , Xiang Li , Jian Yang
show 1 more author
Ying Tai
This is my paper · ORCID
classification cs.CVcs.AI
keywords text-to-videogenerationvideocaptioninginstance-awarecaptionstructuredhallucinationreductionvideo-textdatasetpromptengineeringdiffusiontransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that video captions organized around individual instances, with separate fields for each object's class, appearance, actions, motion, and position, plus background and camera movement, let text-to-video models generate frames closer to the original video and follow fine-grained prompts more accurately. It argues that short captions miss details, dense captions hallucinate, and coarse structured captions stay too vague, so none keep the caption faithful to the video. The authors build a pipeline that isolates instances with an auxiliary model cluster, blurs out non-instance regions, and guides a multimodal LLM with per-class hints and a positive/negative lexicon to produce concise structured phrases. They also curate InstanceVid, 22K videos with these captions, fine-tune Open-Sora on it, and add InstanceEnhancer to turn short user prompts into the same structure at inference. If the claim holds, instance-level structured captions become a practical recipe for higher-fidelity text-to-video training and inference.

What carries the argument

The central machinery is the auxiliary model cluster (AMC) combined with a structured caption schema. AMC uses object detection, video instance segmentation with SAM2, and camera motion detection to transform a global video into local instance videos with blurred backgrounds, so the MLLM annotates each instance in isolation while still receiving the global description as context. The schema fixes a JSON format with 'Global Description' and 'Structured Description' subfields for 'Main Objects' (each with Class, Appearance, Actions and Motion, Position), 'Background Detail', and 'Camera Movement'; the improved Chain-of-Thought prompts, class-specific hints for roughly eighty categories, and a positive/negative lexicon steer the MLLM toward concise, accurate phrases.

What would settle it

Run a perceptual study on the same 100 reconstruction videos where human raters, blind to caption method, pick which regenerated video is closer to the ground truth; if MiraData or dense captions are chosen at least as often as InstanceCap, the 3DV AE advantage would not translate into perceived fidelity. A second check: re-run the Table 2 fine-tuning with InstanceVid captions stripped of their structured JSON into plain paragraphs, and see whether the Inseval gains disappear, which would isolate the structure as the cause.

Watch

Extended reading notes

Core claim

The central claim is that instance-aware structured captions, which decompose a video into a global description plus a structured description of main objects, background, and camera movement, produce higher fidelity to the source video and fewer hallucinations than short captions, dense captions, or coarse structured captions. Quantitatively, the paper reports the best 3DV AE score in reconstruction-via-recaption (134.25 versus 137.50 for MiraData) and higher human-rated instance detail and hallucination scores, while fine-tuning Open-Sora on InstanceVid raises average Inseval accuracy from 28.63% to 37.88%. The authors argue this happens because isolating each instance with blurred backgrounds lets the multimodal LLM describe one object at a time, and because the human-designed class hints and lexical constraints keep those descriptions precise rather than inventive.

Load-bearing premise

The quantitative superiority over MiraData rests on the 3DV AE score, a distance between latent codes of original and regenerated videos whose validation only tested pixel-level distortions, not whether lower scores correspond to captions humans judge as more faithful; the approach also presumes object detection can find the instances that matter, which the paper itself notes fails in instance-free scenes.

Editorial extensions

If this is right

  • Fine-tuning a DiT-based T2V model on instance-aware structured captions improves instance-level instruction following, notably on detail and multi-object prompts, relative to the same base model trained on short, dense, or coarse-structured captions.
  • The 3DV AE metric offers a scalable automatic way to compare caption fidelity without human judges, since it measures latent-space distance between original and recaption-reconstructed videos.
  • The InstanceEnhancer shows that inference-time prompt restructuring can close the gap between short user prompts and long training captions, which is a known weakness of current T2V models.
  • The InstanceVid dataset and the structured caption schema can be reused directly by other T2V training pipelines, since the caption format is model-agnostic.
  • The human-designed class hints and positive/negative lexicon can be extended to new object categories, suggesting the pipeline generalizes beyond the roughly eighty classes tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The blur-background trick, which the paper shows beats red-screen or weak-visual-prompt alternatives for multi-instance scenes, could likely transfer to other MLLM annotation tasks where attribute blending across objects is a known failure mode.
  • If the structured caption format is what drives the gains, one testable extension is to apply InstanceCap-style captions to longer videos or interactive multi-camera footage, where the separation of camera movement from instance action should matter even more.
  • The paper's own stated limitations, that precision depends on object detection and benefits fade in instance-free scenes, suggest the method is strongest for object-centric videos, so its advantage over dense captions may shrink on purely scenic or atmospheric footage.
  • The Inseval benchmark, with its QA-style CoT evaluation, could become a reusable instance-level probe for other T2V models, independent of the captioning method used to train them.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes InstanceCap, a structured captioning framework for text-to-video training data. It uses an auxiliary model cluster (object detection, SAM2 video segmentation, camera-motion detection) to isolate instances, then prompts MLLMs with a chain-of-thought procedure and human-designed class hints to produce a JSON caption containing a global description, per-instance appearance/actions/position, background detail, and camera movement. The authors curate a 22K-video InstanceVid dataset from OpenVid-1M and introduce InstanceEnhancer, a two-stage inference-time prompt enhancement and alignment pipeline. They evaluate by (i) reconstruction-via-recaption with CogVideoX-5b using a 3DV AE distance, CLIP SenbySen, and a user study, and (ii) finetuning Open-Sora on InstanceVid and measuring instance-level success on a new benchmark Inseval. The central claims are that InstanceCap captions are more faithful and less hallucinated than short, dense, or coarse-structured captions, and that finetuning on InstanceVid improves instance-level T2V generation.

Significance. The problem is well chosen: existing recaption paradigms trade off brevity, detail, and structure, and instance-level fidelity is increasingly important for controllable video generation. The instance-decomposition idea with blurred backgrounds and structured JSON output is a sensible and novel way to force MLLMs to attend to individual objects, and the detailed prompts, lexicon, and evaluation protocols are given in the supplement, which is a strength for reproducibility. If the claims hold, the 22K InstanceVid dataset and the Inseval benchmark would be useful resources. The current evidence is suggestive but not conclusive: the reconstruction metric is validated only for pixel-level distortions, and the finetuning result is potentially confounded by the inference-time enhancement pipeline.

major comments (5)
  1. [§3.3, Table 2, Figure S11] The reported gain from finetuning on InstanceVid is confounded with the InstanceEnhancer and Aligning Prompt. The paper does not specify that the base Open-Sora row or the Panda/ShareGPT4Video/LLaVA rows receive the same two-stage expansion and 180-word alignment. Because Inseval prompts are short single sentences, the proposed system may receive much longer and more instance-detailed conditioning, which alone could improve all Inseval dimensions. Please add a matched-prompt control: run InstanceEnhancer/Aligning Prompt on the base model and on each finetuned baseline, or feed the raw Inseval prompt to all rows, and report both settings.
  2. [Supplementary Section 5, Eq. (1), Table S1] The 3DV AEscore is validated only against pixel-level distortions (blur, noise, compression, shifts), not against caption-content differences. The main Table 1 claim requires that this latent distance responds to caption fidelity. Please validate by recaptioning the same videos with captions that are systematically wrong in instance attributes (e.g., swapped colors or actions) and show that 3DV AE separates correct from incorrect captions; also report per-video variance and confidence intervals.
  3. [§4.1, Tables 1 and 2] All reported numbers are point estimates without error bars, significance tests, or the number of independent seeds. The differences in Table 1 are small (3DV AE 134.25 vs. 137.50; CLIP SenbySen 0.2133 vs. 0.2156), and Table 2 reports percentages without stating the number of prompts per cell. Without uncertainty quantification, the claim of significant improvement is not supported. The Figure 8 user study also lacks participant count, video sample size, and inter-annotator agreement.
  4. [Supplementary Section 6, Table 2] Multiple Shape and Multiple Detail are omitted because all models performed poorly, so the reported Average is not an overall average and should be explicitly labeled as the mean over the reported subset. In addition, the Inseval evaluation is an LLM-based judge whose agreement with human preferences is not demonstrated; a small human validation of the QA judge would strengthen the central generation claim.
  5. [§4.3, Table 2] The Open-Sora finetuning setup is not described: learning rate, batch size, number of steps, text-encoder updates, video resolution, and compute are all missing. Since the T2V-generation claim is a major contribution, this missing information prevents replication and makes it difficult to assess whether the gain is due to the InstanceVid data or to other training choices.
minor comments (7)
  1. [Abstract, Table 1] The abstract's claim of significant improvement over previous models is stronger than the evidence: in Table 1, InstanceCap's CLIP SenbySen (0.2133) is below MiraData (0.2156), so the dominance is not across all metrics.
  2. [Figure 8] The Hallucination Scores are shown with higher bars, but the text does not state whether higher is better; please specify the direction and the scale.
  3. [Supplementary Section 5, Eq. (1)] The layer-specific weight matrix w_l is not defined; please clarify whether it is a learned weight matrix or a fixed weighting factor.
  4. [§4.1] Please specify how the 100 reconstruction videos were selected and whether any of them overlap with the source set used to construct InstanceVid.
  5. [Table 2] Use consistent model names ('LLaVA-Video' vs. 'LLaVA') and describe how each '+captioner' row was constructed, including which captions were used for finetuning and which prompt text was used at inference.
  6. [§3.2] The claim that InstanceVid is curated from 'high-aesthetic, high-consistency' videos should cite or describe the filtering criteria, since the selection directly affects the generality of the downstream claims.
  7. [Supplementary Section 2] The statement that full class hints 'will be provided in the code we plan to release later' should be replaced by including the full JSON in the supplement for immediate reproducibility.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation reduces to its inputs; the only circularity-adjacent issue is a mild self-citation in dataset provenance, plus a non-circular confound in the T2V finetuning comparison.

  1. other [Section 3.2 (InstanceVid) and Section 4.1 (Video reconstruction with recaptions)]
    "InstanceVid is curated via refining a subset from the high-aesthetic, high-consistency videos from OpenVid-1M [16]. ... we carefully selected 100 video clips from OpenVid-1M [16] and Animal Kingdom [17]."

    OpenVid-1M is the authors' own prior dataset, and it is used both as the source of the InstanceVid training captions and as part of the reconstruction test set. This is a self-citation/provenance overlap rather than a by-construction equivalence: InstanceCap is not trained to minimize the 3DV AE score, and the test set also includes Animal Kingdom plus a human user study. It mildly weakens the independence of the evaluation but is not load-bearing.

full rationale

The central caption-fidelity claim is supported by an extrinsic reconstruction-via-recaption test: captions from each method are fed into the same CogVideoX-5b model and the latent distance to the original video is measured. No parameter of InstanceCap is fitted to that metric, and the 3DV AE score is a defined distance rather than a trained objective, so the comparison is not circular by construction. The metric's validation covers only pixel-level distortions rather than caption-level differences, which is a validity concern, not a circularity. The T2V finetuning comparison in Table 2 has a genuine confound: the proposed row uses InstanceEnhancer plus the Aligning Prompt at inference, and the paper does not state that baselines receive the same enhanced conditioning. That threatens causal attribution of the finetuning gain, but it is a control issue rather than a prediction that reduces by construction to its inputs. The only self-citation dependency is the OpenVid-1M provenance of InstanceVid and part of the reconstruction test set; because the evaluation also uses Animal Kingdom and a human user study, and because the method is not optimized against the evaluation metrics, this does not make the result forced. Overall, the paper's claims are not circular, but the mild self-citation and dataset overlap justify a low non-zero score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on no mathematical derivation; instead it depends on a chain of modeling assumptions: the auxiliary model cluster produces accurate instance masks, the MLLM with blur-background inputs and class hints produces truthful captions, and the two evaluation metrics (3DV AE score and CLIP SenbySen) capture caption fidelity. The hand-curated lexicon and class hints are author-designed inputs that are not independently validated. No new physical or conceptual entities are introduced.

free parameters (3)
  • InstanceVid dataset scale = 22K videos
    The finetuning result (Table 2) depends on this curated scale; the number is a budget/availability choice, not derived.
  • Number of human-designed class hints = ~80 classes
    The class prompts are hand-authored (Supplement Section 2); the MLLM outputs depend on this prompt set.
  • Evaluation set size for reconstruction = 100 clips
    The headline 3DV AE score (Table 1) is computed on 100 manually selected clips; the choice of clips is a design decision that could affect scores.
assumptions (5)
  • domain assumption CLIP text-image similarity is a valid proxy for video-text semantic alignment.
    Used in CLIP SenbySen (Eq. 2) for reconstruction evaluation; no human validation of the sentence-frame similarity thresholds is provided.
  • domain assumption 3DV AE latent distance captures perceptual fidelity of reconstructed videos.
    Supplementary Eq. 1; validation in Table S1 uses pixel-level distortions, not caption-induced differences, so correlation with human perception of caption fidelity is assumed.
  • domain assumption CogVideoX-5b is a faithful regenerator whose outputs can be compared to ground truth.
    Section 4.1 uses CogVideoX-5b for reconstruction; stochasticity and prompt-following biases are not modeled.
  • domain assumption The auxiliary model cluster (detector, SAM2, camera motion detector) provides accurate instance masks.
    Section 3.1 decomposes global videos into local instances using these models; mask errors propagate into the captions, as the paper's own Limitations section concedes.
  • ad hoc to paper The hand-curated Positive/Negative Lexicon and class hints improve captions without introducing bias.
    These are author-designed lexical restrictions (Supplement Figure S1) applied at annotation and inference time; their aesthetic bias is not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption." pith.science (2026). https://pith.science/paper/GDU2EBDO

@misc{pith2026241209283,
  author       = {Pith},
  title        = {Pith review of: InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDU2EBDO}},
  note         = {Machine review of arXiv:2412.09283}
}
read the original abstract

Text-to-video generation has evolved rapidly in recent years, delivering remarkable results. Training typically relies on video-caption paired data, which plays a crucial role in enhancing generation performance. However, current video captions often suffer from insufficient details, hallucinations and imprecise motion depiction, affecting the fidelity and consistency of generated videos. In this work, we propose a novel instance-aware structured caption framework, termed InstanceCap, to achieve instance-level and fine-grained video caption for the first time. Based on this scheme, we design an auxiliary models cluster to convert original video into instances to enhance instance fidelity. Video instances are further used to refine dense prompts into structured phrases, achieving concise yet precise descriptions. Furthermore, a 22K InstanceVid dataset is curated for training, and an enhancement pipeline that tailored to InstanceCap structure is proposed for inference. Experimental results demonstrate that our proposed InstanceCap significantly outperform previous models, ensuring high fidelity between captions and videos while reducing hallucinations.

Figures

Figures reproduced from arXiv: 2412.09283 by the authors.

Figure 1
Figure 1. Top: Comparison of the reconstruction-via-recaption results between InstanceCap and state-of-the-art captioning methods for annotating the ground truth video. InstanceCap produces results that more closely resemble the original video, showing greater detail fidelity (highlighted by the red circle). Bottom: The corresponding captions generated by InstanceCap and others. Red denotes incorrect captions, blue represents… view at source ↗
Figure 2
Figure 2. Overview of InstanceCap pipeline. Details of “from dense prompts to structured phrases” design are shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Details on “from dense prompts to structured phrases” design. We propose an improved CoT pipeline with carefully designed information interactions (red arrow), which facilitates MLLMs to accurately capture instances with precise descriptions on attributes. based video recaption methods shows that simple prompts, like “Please provide a detailed description of this video” or Chain-of-Thought prompts “Let’s think step … view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: High-level overview of InstanceEnhancer, illustrating [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Comparison on reconstruction-via-recaption be [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Visual comparison of InstanceCap and Opensora on Single and Multiple Action Score. In terms of the dynamic degree of video generation, we show better consistency and enhanced multi-instance dynamic generation effect. and compute CLIP [19] similarity between each senten…
Figure 8
Figure 8. Figure 8: User study on instance detail and hallucination scores. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Visual comparison of InstanceCap and Open-Sora on instance-level attributes. InstanceCap excels in precise in￾stance detail fidelity and instruction-following capabilities, even with complex multi-instance and multi-attribute scenarios. Sora [35]. In [PITH_FULL_IMAGE:…
Figure 10
Figure 10. Figure 10: (a) Ablation study on the effect of camera movement hints on the accuracy of MLLM labeling. (b) Impact of human-designed [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: (a) Comparison against the weak visual prompt for reconstruction-via-caption visualization on multi-instance targets. (b) [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 46 canonical work pages

  1. [1]

    Chen and William B

    David L. Chen and William B. Dolan. Collecting highly par- allel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL-2011), Portland, OR, 2011. 2

    work page 2011

  2. [2]

    VideoCrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafter2: Overcoming data limitations for high-quality video diffusion models. In CVPR, pages 7310–7320, 2024. 1, 3

    work page 2024

  3. [3]

    Sharegpt4video: Improving video understand- ing and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 1, 3, 5, 7, 2

    arXiv 2024

  4. [4]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:2402.19479 ,

    arXiv

  5. [5]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xi- hui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. arXiv preprint arXiv:2307.06350, 2023. 6

    arXiv 2023

  6. [6]

    VBench: Com- prehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Com- prehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Reco...

    work page 2024

  7. [7]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023. 4

    work page 2023

  8. [8]

    Pyramidal flow matching for efficient video generative modeling

    Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954 ,

    arXiv

Show all 60 references
  1. [9]

    Miradata: A large-scale video dataset with long durations and structured captions

    Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xin- tao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407.06358 ,

    arXiv

  2. [10]

    Kuaishou. Kling. https://kling.kuaishou.com ,

  3. [11]

    Open-sora-plan, 2024

    PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, 2024. 3, 6, 7

    2024

  4. [12]

    T2v- turbo-v2: Enhancing video generation model post-training through data, reward, and conditional guidance design.arXiv preprint arXiv:2410.05677, 2024

    Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, and William Yang Wang. T2v- turbo-v2: Enhancing video generation model post-training through data, reward, and conditional guidance design.arXiv preprint arXiv:2410.05677, 2024. 1

  5. [13]

    Evaluating text-to-visual generation with image-to-text gen- eration, 2024

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text gen- eration, 2024. 6

    2024

  6. [14]

    Evalcrafter: Benchmarking and evaluating large video generation models

    Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Ray- mond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. arXiv preprint arXiv:2310.11440, 2023. 6, 2

    arXiv

  7. [15]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 1

    2024 arXiv

  8. [16]

    Openvid-1m: A large-scale high-quality dataset for text-to- video generation

    Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhen- heng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to- video generation. arXiv preprint arXiv:2407.02371, 2024. 3, 4, 5, 7

    2024 arXiv

  9. [17]

    Animal kingdom: A large and diverse dataset for animal behavior understanding

    Xun Long Ng, Kian Eng Ong, Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu. Animal kingdom: A large and diverse dataset for animal behavior understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19023–19034, 2022. 5

    2022

  10. [18]

    Pika 1.0

    Pika. Pika 1.0. https://pika.art, 2023. 1, 3

    2023

  11. [19]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 6, 2

    2021

  12. [20]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

    2024 arXiv

  13. [21]

    Runway. Gen-2. https://research.runwayml. com/gen2, 2023. 1

    2023

  14. [22]

    What does clip know about a red circle? vi- sual prompt engineering for vlms

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? vi- sual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11987–11997, 2023. 3, 8

    2023

  15. [23]

    ModelScope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. ModelScope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023. 1

    2023 arXiv

  16. [24]

    Vatex: A large-scale, high- quality multilingual dataset for video-and-language research,

    Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high- quality multilingual dataset for video-and-language research,

  17. [25]

    LaVie: High-quality video gener- ation with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. LaVie: High-quality video gener- ation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023. 1

    2023 arXiv

  18. [26]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In The Twelfth International Conference o...

    2023

  19. [27]

    Ifadapter: Instance feature control for grounded text-to-image generation, 2024

    Yinwei Wu, Xianpan Zhou, Bing Ma, Xuefeng Su, Kai Ma, and Xinchao Wang. Ifadapter: Instance feature control for grounded text-to-image generation, 2024. 6

    2024

  20. [28]

    Vript: A video is worth thousands of words, 2024

    Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words, 2024. 3

    2024

  21. [29]

    Mastering text-to-image dif- fusion: Recaptioning, planning, and generating with multi- modal llms

    Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Ste- fano Ermon, and CUI Bin. Mastering text-to-image dif- fusion: Recaptioning, planning, and generating with multi- modal llms. In Forty-first International Conference on Ma- chine Learning, 2024. 5

    2024

  22. [30]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 3, 5, 6, 7, 2

    2024 arXiv

  23. [31]

    Cpt: Colorful prompt tuning for pre-trained vision-language models

    Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat- Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. AI Open, 5:30–38,

  24. [32]

    Show-1: Marrying pixel and latent diffusion models for text-to-video generation

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. IJCV, 2024. 1

    2024

  25. [33]

    Efros, Eli Shecht- man, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric, 2018. 2

    2018

  26. [34]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. 3

    2024

  27. [35]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 3, 6, 7

    2024

  28. [36]

    Open-Sora: Democratizing efficient video production for all

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing efficient video production for all. https://github.com/hpcaitech/Open- Sora, 2024. 1

    2024

  29. [37]

    Luowei Zhou, Nathan Louis, and Jason J. Corso. Weakly- supervised video object grounding from text by loss weight- ing and object interaction, 2018. 2

    2018

  30. [38]

    Detrs with col- laborative hybrid assignments training

    Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with col- laborative hybrid assignments training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6748–6758, 2023. 3 10 InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured...

    2023

  31. [39]

    Conversely, we manually constructed a Negative Lexicon, which was further enriched using the powerful LLM, GPT-4o

    Positive/Negative Lexicon To enhance the aesthetic quality of generated videos, we carefully collected prompts from various open-source model galleries, extracting adjectives to build a Positive Lexicon. Conversely, we manually constructed a Negative Lexicon, which was further...

  32. [40]

    Please describe the car by its color, make, model, condition, license plate (if visible), and any distinguishing features such as stickers, dents, or modifications

    Human-designed Class Hints For the Human-designed Class Hints, we carefully crafted additional prompts for over eighty categories, each specif- ically tailored to its specific characteristics. Below, we present twenty of these categories. The full JSON- formatted hints for all...

  33. [41]

    Please describe this video in one sentence, no more than 20 words

    Prompt Design of Figure 3 System prompt. Referring to ShareGPT4Video [3], we divided the System prompt into three parts. Through exten- sive tests on challenging samples, including multi-instance, complex scenes, and high-intensity motion, we finalized the system prompt shown ...

  34. [42]

    To pro- vide more precise instructions to LLMs, we meticulously designed multiple examples as part of the CoT, which are fed into the LLMs

    Design of InstanceEnhancer In InstanceEnhancer, prompt alignment during inference is achieved through a two-stage process (Figure S2). To pro- vide more precise instructions to LLMs, we meticulously designed multiple examples as part of the CoT, which are fed into the LLMs. An...

  35. [43]

    subject" +

    Evaluation metrics for video reconstruction 3DV AE score (3DV AEscore). The LIPIPS score [33] which is widely used to evaluate image reconstruction qual- ity, measures perceptual distance between ground truth (GT) and reconstructed images. We extent this concept for video data...

  36. [44]

    In the vast whiteness of the snowy forest, two wolves stalk quietly…

  37. [45]

    On the bank of a quiet river, two men sit, fishing rods in hand, lost in their shared pursuit…

  38. [46]

    The film opens with a solitary tree standing tall in the middle of a vast field, perfectly motionless… [Other examples] [Example B]

  39. [47]

    a wolf BREAK a wolf BREAK a rabbit

  40. [48]

    a man BREAK a man BREAK a woman

  41. [49]

    a tree BREAK a boy BREAK a man BREAK a woman [Other examples] [Example]

  42. [50]

    Two wolves were hunting a rabbit in the snow

  43. [51]

    Two men were fishing and a woman was standing by

  44. [52]

    Global Description

    A completely still tree. [Other examples] { "Global Description": "The astronaut is riding a horse.", "Structural Description": { "Main Instance": { "No.0": { "Class": "an astronaut", "Appearance": "An astronaut, clad in a sleek and pristine spacesuit with a shimmering...", "A...

  45. [53]

    3) Ex- trinsic Hallucination: Evaluate whether the text introduces content that is not present in the video

    Intrinsic Hallucination: Evaluate whether the text hallu- cinates descriptions of things present in the video. 3) Ex- trinsic Hallucination: Evaluate whether the text introduces content that is not present in the video. For convenience, the latter two have been combined into a...

  46. [54]

    counter-intuitive

    Inseval Inference prompts of Inseval. In implementing Inseval, we designed multiple prompts to test each dimension, as illustrated in Figure S3. To further evaluate the model’s generative capabilities and instruction-following accuracy, 4 Instance Detail Hallucination Scores 1...

  47. [55]

    ,".join([f

    Analysis on Commercial Products vs. Open- source Models Prompt processing analysis. Commercial T2V products excel at processing complex input prompts, effectively han- dling long-form text in structured formats while preserving semantic coherence. They can seamlessly interpret...

  48. [56]

    subject" +

    Please use the "subject" + "attribute" + "position" structure more often. For example, "An old man, his hair is white."

  49. [57]

    The video shows

    Please tell me the content of the video directly, don't use "The video shows..." Or other similar forms, you should begin with a direct description of the content, for example:"An old man..."

  50. [58]

    A man... and a woman..., and a man

    If there are multiple objects, such as people, introduce them with phrases like "A man... and a woman..., and a man...","A car..., and a car..."

  51. [59]

    The scene is

    When you need to describe The background in detail, use "The scene is..." As an opening sentence

  52. [60]

    Aligning prompt used during alignment with the open source model

    Summarize it to approximately 180 words Figure S11. Aligning prompt used during alignment with the open source model. 9

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.