Pith. sign in

REVIEW 4 major objections 7 minor 18 cited by

ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One-pass video customization keeps multiple identities apart

desk verdict ConceptMaster is a serious engineering contribution to multi-concept video customization, but its decoupling metrics rest on a LISA training/evaluation loop and need independent validation. read the letter →

arxiv 2501.04698 v2 pith:FXOWWXCD submitted 2025-01-08 cs.CV

Yuzhou Huang , Ziyang Yuan , Quande Liu , Qiulin Wang , Xintao Wang , Ruimao Zhang , Pengfei Wan , Di Zhang
show 1 more author
Kun Gai
This is my paper · ORCID
classification cs.CV
keywords multi-conceptvideocustomizationidentitydecouplingdiffusiontransformerdecoupledconceptembeddingszero-shotpersonalizationgenerationdataconstructionpipelineDecoupleAttentionModule
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ConceptMaster aims to solve multi-concept video customization: given several reference images and a caption, generate a video in which each depicted identity matches its reference while the identities stay visually distinct from one another. Existing approaches either require per-concept test-time tuning or blend attributes together when several concepts appear at once, especially when the concepts look alike. The paper argues that the fix is to learn each concept as its own multimodal embedding and inject all of them through a dedicated cross-attention layer that does not disturb the caption-conditioning path. It also addresses the data shortage by building a pipeline that extracts over 1.3 million video-entity pairs with per-entity images and labels, and it evaluates on a 210-sample set spanning six composition scenarios. If correct, this makes feed-forward multi-concept personalization practical for video generation.

What carries the argument

The load-bearing object is the decoupled composite multi-concept embedding $\mathbf{c}^*_{\text{IDs}} = \mathrm{Concat}(c_1,\dots,c_N)$, produced by three components: a learnable Q-Former that turns CLIP's dense image tokens into compact visual embeddings; a Decouple Attention Module that performs cross-attention between each visual embedding and its own text label embedding so that each concept's representation is bound to its semantic identity; and a Multi-Concept Injector, a dedicated cross-attention layer placed after the original text cross-attention in each transformer block, which consumes the composite as keys and values. The standalone injector is what lets the model inject many identities without overwriting the caption path.

What would settle it

Take a test set with two visually similar concepts, generate videos with ConceptMaster, and have annotators or an independent segmentation tool, not the one used in training, mark each concept's pixels; if identity swap rates or CLIP-I and DINO-I values on those masks are close to baseline, the decoupling advantage is an artifact of the metric's segmentor.

Watch

Extended reading notes

Core claim

The paper's central claim is that identity decoupling in multi-concept video customization reduces to representing each concept by a visual-text pair and injecting the concatenation of those paired embeddings through a standalone cross-attention layer, rather than merging them into the text-conditioning stream. Concretely, CLIP dense visual tokens are compressed by a Q-Former, a Decouple Attention Module binds each visual embedding to its own textual label, and a Multi-Concept Injector cross-attention layer inside every transformer block consumes the composite embedding as keys and values. This keeps the original text cross-attention intact, which the paper presents as the reason caption semantics stay aligned while multiple identities, including visually similar ones, remain separated. The paper reports that this design outperforms two-stage pipelines and alternative injection schemes on concept fidelity, identity decoupling, and video quality.

Load-bearing premise

The claim that identities are truly decoupled rests on automated scores computed from masks produced by the same segmentation model that was used to construct the training data, so the measured separation could partly reflect what that segmentor finds easy to segment.

Editorial extensions

If this is right

  • Users can specify several subjects from reference images and get customized videos in a single forward pass, removing the per-concept optimization step.
  • Visually similar concepts remain separable, so the method can handle cases like two people or two animals of the same category without attribute blending.
  • Because the original text cross-attention is untouched, the caption keeps guiding action and scene even while identity conditions are injected.
  • The 1.3M-pair data pipeline offers a reusable recipe for collecting multi-concept video-entity training data at scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same standalone-injector idea to single-concept personalization or to image generation, where decoupling pressure is lower; the paper does not test this.
  • The decoupling metric depends on the same segmentation model used to build training data, so an independent human-annotation study or a different segmentor would show whether the reported decoupling numbers reflect true visual separation.
  • The design suggests that composing many conditions, such as identity, style, and layout, via separate cross-attention layers could be a general scaling recipe for transformer-based generators, though the paper only demonstrates identity conditions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces ConceptMaster, a feed-forward framework for multi-concept video customization without test-time tuning. Given a video caption, a set of reference images, and a text label per concept, ConceptMaster extracts dense CLIP visual tokens, compresses them with a learnable Q-Former, binds each visual embedding to its corresponding label via a Decouple Attention Module (DAM), concatenates the per-concept embeddings, and injects the composite representation into a transformer-based text-to-video diffusion model through an additional standalone cross-attention layer (MC-Injector). To support training, the authors construct a 1.3M video-entity dataset using a pipeline that combines video captioning, noun extraction, Grounding-DINO, CLIP classification, LISA segmentation, and FaceAnalysis. They evaluate on a manually collected set of 210 videos across six concept-composition scenarios, reporting improvements over four multi-concept image customization methods combined with an image-to-video model on CLIP-based and DINO-based identity metrics, decoupling metrics, and VBench-style video quality metrics.

Significance. If the evaluated results are reliable, the paper makes a practical contribution: a single-pass, tuning-free method for multi-concept video customization, together with a large-scale data collection pipeline that is human-verified to improve extraction success rates over a Grounded-SAM baseline. The architectural choices (intra-pair text-conditioned attention in DAM and a separate cross-attention injector) are reasonable and are ablated against plausible alternatives, which is a strength. The feed-forward inference and the scale of the dataset are also positive attributes. However, the significance is conditional on addressing evaluation weaknesses: the decoupling metrics rely on the same segmentor used in training, the comparison confounds the customization method with the underlying video backbone, and no statistical support is provided for the reported margins.

major comments (4)
  1. [Section 5.1, Section 4.3] The decoupling metrics (CLIP-I, DINO-I, CLIP-tag, CLIP-tagdis) are computed on masks generated by LISA, and LISA is also the core segmentor in the training-data construction pipeline. This shared-tool choice can bias the decoupling measurement: the model may learn to generate videos that are easy for LISA to segment, inflating the reported CLIP-I/DINO-I margins over the baselines without verified visual identity separation. The paper provides no human evaluation of identity separation, no manual verification of LISA masks on generated videos, and no alternative segmenter. Please add (a) a manual mask-quality check on a subset of generated videos, (b) the same metrics computed with a different segmenter (e.g., SAM prompted with boxes) or with human-drawn masks, and (c) a human perceptual study on identity decoupling for the 'highly similar concepts' cases. This concern is load-bearing because the central novelty of the paper is decoupling.
  2. [Section 5.3, Table 1] The comparison is not controlled for the underlying video generation backbone. All baselines are two-stage pipelines that use I2VGen-XL after a multi-concept image customization model, whereas ConceptMaster uses its own proprietary transformer-based T2V model. Consequently, the video-quality differences (Motion Smoothness, Dynamic Degree, Aesthetic Quality, Imaging Quality) and even CLIP-cap may stem from the base model rather than from the proposed customization method. Please add a controlled setting (e.g., feeding the same base model with image conditions, or comparing ConceptMaster against itself without the MC-Injector) or at minimum explicitly discuss this confound and provide the text-only T2V output of the base model as a reference. Without such a control, the claim of significantly outperforming previous methods in video quality is overstated.
  3. [Section 5.2, Table 1] All quantitative results are single means over only 210 manually built samples, with no standard deviations, confidence intervals, or significance tests. Several reported differences are small (e.g., CLIP-tag 22.165 vs 21.980 and CLIP-tagdis 16.169 vs 16.325), so the reader cannot judge whether the advantages are robust. Please report per-sample distributions, per-scenario results, and paired significance tests (e.g., bootstrap or Wilcoxon signed-rank), and also break out the 'highly similar concepts' subset that the abstract emphasizes. This is important because the headline claim is statistical in nature.
  4. [Section 4.3, Supplementary Section 3] The data pipeline includes a human-verified success-rate comparison against Grounded-SAM on 2000 Panda-2M samples, which is a strength. However, the causal contribution of the pipeline to generation quality is not demonstrated. Please add an ablation that trains the same ConceptMaster architecture on data produced by a simpler Grounded-SAM-based pipeline versus the proposed pipeline, reporting the same evaluation metrics. This would substantiate the claim that the proposed data collection addresses the scarcity problem rather than merely being a large-scale engineering effort.
minor comments (7)
  1. [Section 4.2, Eq. (2)] The activation function is written as 'GLUE' with reference [57], which is a benchmark paper; this appears to be a typo for GELU or GLU, so please correct the name and cite the appropriate source.
  2. [Section 4.3] The phrase 'filtering out low-quality videos that are not unsuitable for the task' contains a double negative and should read 'that are unsuitable for the task'.
  3. [Section 4.1] The phrase 'the paire images and label for each concept' contains a typo; it should be 'the paired images and label'.
  4. [Table 4] The caption template 'staying besides' should be 'staying beside'.
  5. [Section 5.1] The metrics CLIP-cap, CLIP-tag, and CLIP-tagdis are not precisely defined; please specify which CLIP model and prompt templates are used and how the mask-image similarities are aggregated across frames.
  6. [Conclusion] The paper would benefit from a brief limitations paragraph discussing the taxonomy's coverage, the dependence on LISA/CLIP in data construction and evaluation, and the scale of the evaluation set.
  7. [Reproducibility] Since the base text-to-video model is proprietary, please provide code or detailed training configuration, or release a subset of the evaluation set and masks, to allow independent verification of the quantitative claims.

Circularity Check

1 steps flagged · score 3.0 of 10

The framework and fidelity claims are empirically supported, but the central identity-decoupling metric reuses LISA, the same segmentor used to construct the training masks, so the decoupling measurement is not independent of the training signal.

  1. fitted input called prediction [Sec. 4.3 (data construction) and Sec. 5.1 (evaluation metrics), with results in Table 1]
    "Training: "we employ the same frame sampling strategy and use LISA [33], an MLLM-based [39] segmentor, input by both text prompts and images with strong visual reasoning capabilities, to extract entity masks." Evaluation: "Decoupling ability, where we utilize LISA [33] to segment the mask area of each concept in generated videos, and then compute CLIP-I and DINO-I [4] scores between the original concept images and the mask areas of each concept in generated videos.""

    The same segmentation model, LISA, defines the entity regions in the training data and defines the entity regions in the generated videos used to compute the decoupling scores. A model trained on LISA-derived entity masks can therefore improve CLIP-I, DINO-I, CLIP-tag, and CLIP-tagdis by producing videos that LISA segments cleanly into the prompted concept classes, even if a human or an independent segmenter would not perceive the identities as cleanly separated. The paper reports no human verification of masks on generated videos and no second segmentor, so the central claim of identity decoupling rests on a metric whose instrument is shared with the training-data construction pipeline.

full rationale

The paper's main contribution is an empirical video-customization system, not a formal derivation, and most of its evaluation is externally grounded: the 210-sample evaluation set is manually collected to avoid leakage, the baselines in Tables 1 and 2 are reimplemented or open-sourced methods, and the ablations in Table 3 compare architectural variants trained on the same data, so the concept-fidelity and generation-quality claims are not forced by construction. No load-bearing uniqueness theorem is imported from the authors' prior work, and the self-citations that appear (e.g., ID-Animator, DynamicCrafter, PhotoMaker, VideoCrafter) are contextual references rather than justifications of the central design. The only material circularity concern is the repeated use of LISA: LISA builds the training entity masks in Sec. 4.3, and LISA also produces the masks in generated videos on which the decoupling metrics in Sec. 5.1 are computed. Because the paper reports no human or independent-segmentor verification of those generated-video masks, the decoupling advantage in Table 1 may partly reflect alignment with LISA's segmentation preferences rather than independently verified identity separation. This is a genuine measurement confound in the paper's central claim, but it is not a strict case of a prediction being identical to a fitted input, so a score of 3 is appropriate rather than a higher score for full by-construction circularity.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The load-bearing assumptions are domain assumptions about the sufficiency of CLIP-based features, the accuracy of the automated data pipeline, and the validity of LISA-based evaluation. Free parameters are training and inference hyperparameters chosen by hand, plus the evaluation set design. The main risk lies in the shared use of LISA for data construction and evaluation, which can bias decoupling measurements.

free parameters (7)
  • Learning rate = 5e-6
    Chosen by hand for Adam optimizer; not derived from data or theory.
  • Global batch size = 256
    Training setup; selected to fit 64 H800 GPUs.
  • CFG scale = 7.5
    Inference hyperparameter; typical range for diffusion guidance.
  • DDIM steps = 100
    Inference hyperparameter; no sensitivity analysis provided.
  • Drop probabilities for CFG = caption 50%, reference 33%
    Classifier-free guidance dropout rates; chosen by hand.
  • Data sampling ratio = 8:1:1
    Ratio of MCVC data, BLIP-Diffusion, and CelebV; chosen by hand.
  • Number of concept classes in taxonomy = 120
    Taxonomy of 120 classes with sub-words for noun extraction; arbitrary granularity.
assumptions (4)
  • domain assumption CLIP image encoder features, after Q-Former processing, retain sufficient identity information for arbitrary reference concepts.
    Sec 4.2; if CLIP/Q-Former lose identity details for visually similar concepts, decoupling fails.
  • domain assumption The automated data pipeline (Qwen2-VL, Grounding-DINO, LISA, CLIP filtering) produces high-quality MCVC pairs at scale without human verification of each pair.
    Sec 4.3; the model's capability depends on this training data being accurate.
  • domain assumption LISA can accurately segment the generated video's concept regions for evaluation, so CLIP-I and DINO-I decoupling scores are meaningful.
    Sec 5.1; if LISA fails on generated content, decoupling metrics are unreliable; LISA is also used in data construction.
  • standard math The base transformer video diffusion model with flow matching (Eq. 1) is a competent backbone that can be adapted by fine-tuning all but 3D self-attention.
    Standard diffusion and flow matching framework; no proof required.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning." pith.science (2026). https://pith.science/paper/FXOWWXCD

@misc{pith2026250104698,
  author       = {Pith},
  title        = {Pith review of: ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FXOWWXCD}},
  note         = {Machine review of arXiv:2501.04698}
}
read the original abstract

Text-to-video generation has made remarkable advancements through diffusion models. However, Multi-Concept Video Customization (MCVC) remains a significant challenge. We identify two key challenges for this task: 1) the identity decoupling issue, where directly adopting existing customization methods inevitably mix identity attributes when handling multiple concepts simultaneously, and 2) the scarcity of high-quality video-entity pairs, which is crucial for training a model that can well represent and decouple various customized concepts in video generation. To address these challenges, we introduce ConceptMaster, a novel framework that effectively addresses the identity decoupling issues while maintaining concept fidelity in video customization. Specifically, we propose to learn decoupled multi-concept embeddings and inject them into diffusion models in a standalone manner, which effectively guarantees the quality of customized videos with multiple identities, even for highly similar visual concepts. To overcome the scarcity of high-quality MCVC data, we establish a data construction pipeline, which enables collection of high-quality multi-concept video-entity data pairs across diverse scenarios. A multi-concept video evaluation set is further devised to comprehensively validate our method from three dimensions, including concept fidelity, identity decoupling ability, and video generation quality, across six different concept composition scenarios. Extensive experiments demonstrate that ConceptMaster significantly outperforms previous methods for video customization tasks, showing great potential to generate personalized and semantically accurate content for video diffusion models.

Figures

Figures reproduced from arXiv: 2501.04698 by the authors.

Figure 1
Figure 1. We propose ConceptMaster, a Multi-Concept Video Customization (MCVC) method that can create high-quality concept [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Directly applying single-concept method cannot han [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of ConceptMaster framework. Given a caption along with a set of concept images and their semantic labels, we [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: (a) The overview of multi-concept data collection pipeline. When dealing with complex scenarios that contain concepts with [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison on multi-concept customization. When compared to several different methods to conduct the MCVC [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Different injection methods of multi-concept references. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Overview framwork of our base text-to-video generation [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: More Qualitative comparison on multi-concept customization between ConceptMaster and naively combining the multi-concept [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: More Qualitative comparison on different injection methods of multi-concept references. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: More Qualitative comparison the effectiveness of the Q-Former and DAM modules. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: More qualitative results of ConceptMaster on diverse scenarios (1/2). [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: More qualitative results of ConceptMaster on diverse scenarios (2/2). [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ID-V2V: Identity-Preserving Video Restylization

    cs.CV 2026-07 conditional novelty 6.0 of 10

    ID-V2V restyles video by conditioning a diffusion model on edited keyframes, depth, relit faces, and face normals, so scene edits propagate while facial identity and performance are preserved.

  2. HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enhancement

    cs.CV 2026-07 conditional novelty 6.0 of 10

    HOMIE unifies inter- and intra-subject video personalization by injecting MLLM-derived relational features into DiT self-attention (GMG) and tagging tokens with modality/reference embeddings (MRE), reporting SOTA on a...

  3. Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Aura combines VLM meta-queries, T5-teacher alignment, subject-aware RoPE shifts, memory tokens, and a large AIGC-curated dataset to claim SOTA multi-element subject-to-video generation under OpenS2V-Eval Total score.

  4. RefAlign: Representation Alignment for Reference-to-Video Generation

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Explicit training-time alignment of DiT reference features to a VFM (with pull/push loss) raises OpenS2V-Eval TotalScore over prior R2V methods with no inference cost.

  5. OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model

    cs.SD 2026-02 conditional novelty 6.0 of 10

    A zero-shot model that generates a video of a reference face speaking user-chosen text with a reference voice timbre.

  6. TinyHistory: Lightweight Video History Embeddings via Two-Stage Context Learning

    cs.CV 2025-12 conditional novelty 6.0 of 10

    TinyHistory compresses long video history into a ~5k-token context via a two-stage learning scheme, achieving consistency on par with heavier baselines at lower memory cost.

  7. UniVideo: Unified Understanding, Generation, and Editing for Videos

    cs.CV 2025-10 conditional novelty 6.0 of 10

    UniVideo combines a frozen MLLM and a video DiT to unify video understanding, generation, in-context editing, visual prompting, and zero-shot free-form video edits under one instruction interface.

  8. Phantom-Data : Towards a General Subject-Consistent Video Generation Dataset

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Phantom-Data provides around one million cross-context, identity-consistent reference-video pairs for subject-to-video generation, and training on it improves prompt following and visual quality.

  9. DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformers

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A diffusion transformer model generates human-product demonstration videos from paired human and product images while preserving both identities through masked cross-attention and motion template guidance.

  10. PolyVivid: Vivid Multi-Subject Video Generation with Cross-Modal Interaction and Enhancement

    cs.CV 2025-06 conditional novelty 6.0 of 10

    PolyVivid combines VLLM-based grounding, 3D-RoPE positional encoding, and attention-inherited identity injection to generate customized videos with multiple consistent subjects and text-specified interactions.

  11. UNIC: Unified In-Context Video Editing

    cs.CV 2025-06 conditional novelty 6.0 of 10

    One diffusion transformer handles ID insert, swap, delete, stylization, propagation, and re-camera control in a single model using in-context token concatenation with task-aware positional encoding and bias.

  12. AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    AnimeShooter provides hierarchical story and shot annotations plus reference images for 148K one-minute animation stories, and AnimeShooterGen trained on it shows improved cross-shot consistency.

  13. OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A benchmark and five-million-clip dataset for evaluating and training subject-to-video generation models, with three new metrics for subject consistency, naturalness, and text alignment.

  14. Vera: Identity-Faithful Human Subject-to-Video Generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Vera improves identity consistency in human subject-to-video generation using cross-clip identity-aligned data, face-weighted masked loss, and layer-aware reference attention.

  15. From Black Box to Transparency: Enhancing Automated Interpreting Assessment with Explainable AI in College Classrooms

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    An explainable model using BLEURT, CometKiwi, pause features, and Chinese phraseological diversity predicts human-rated quality dimensions in English-Chinese consecutive interpreting, with SHAP identifying the stronge...

  16. Tora2: Motion and Appearance Customized Diffusion Transformer for Multi-Entity Video Generation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Tora2 adds decoupled personalization embeddings, gated self-attention binding, and contrastive learning to Tora, enabling simultaneous appearance and trajectory customization for multiple entities in generated video.

  17. Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation

    cs.CV 2025-08 conditional novelty 4.0 of 10

    A hierarchical direct preference optimization with four alignment levels plus automated data selection improves physical plausibility of text-to-video models.

  18. A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Quality

    cs.CV 2025-07 conditional novelty 3.0 of 10

    A survey of 32 long-video generation papers, presenting a taxonomy and component recommendations for backbones, text encoders, objectives, and positional encodings.

Reference graph

Works this paper leans on

95 extracted references · 38 canonical work pages · cited by 18 Pith papers

  1. [1]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1728–1738,

  2. [2]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 2

    2023

  3. [3]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  4. [4]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 6

    2021

  5. [5]

    PySceneDetect

    Brandon Castellano. PySceneDetect. 5

  6. [6]

    Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7310– 7320, 2024. 2, 3

    2024

  7. [7]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13320–13331, 2024. 5, 2

    2024

  8. [8]

    Anydoor: Zero-shot object-level im- age customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level im- age customization. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6593–6602, 2024. 2, 3

    2024

Show all 95 references
  1. [9]

    Custom-edit: Text-guided image edit- ing with customized diffusion models

    Jooyoung Choi, Yunjey Choi, Yunji Kim, Junho Kim, and Sungroh Yoon. Custom-edit: Text-guided image edit- ing with customized diffusion models. arXiv preprint arXiv:2305.15779, 2023. 2

    2023 arXiv

  2. [10]

    Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution

    Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdul- mohsin, et al. Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neu...

    2024

  3. [11]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011, 2023. 2

    2023 arXiv

  4. [12]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  5. [13]

    An image is worth one word: Personalizing text-to- image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patash- nik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to- image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022. 2, 3

    2022 arXiv

  6. [14]

    Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023

    Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023. 2, 3

    2023

  7. [15]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 2, 3

    2023 arXiv

  8. [16]

    Svdiff: Compact pa- 9 rameter space for diffusion fine-tuning

    Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact pa- 9 rameter space for diffusion fine-tuning. arXiv preprint arXiv:2303.11305, 2023. 2

    2023 arXiv

  9. [17]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5

    2016

  10. [18]

    Id-animator: Zero-shot identity-preserving human video generation.arXiv preprint arXiv:2404.15275, 2024

    Xuanhua He, Quande Liu, Shengju Qian, Xin Wang, Tao Hu, Ke Cao, Keyu Yan, Man Zhou, and Jie Zhang. Id-animator: Zero-shot identity-preserving human video generation.arXiv preprint arXiv:2404.15275, 2024. 2, 3

    2024 arXiv

  11. [19]

    Latent video diffusion models for high-fidelity video generation with arbitrary lengths

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221, 2022. 2

    2022 arXiv

  12. [20]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 1

    2022 arXiv

  13. [21]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 3

    2022 arXiv

  14. [22]

    spaCy: Industrial-strength Natural Lan- guage Processing in Python

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength Natural Lan- guage Processing in Python. 2020. 6

    2020

  15. [23]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 2

    2021 arXiv

  16. [24]

    Story3d-agent: Exploring 3d storytelling visualization with large language models

    Yuzhou Huang, Yiran Qin, Shunlin Lu, Xintao Wang, Rui Huang, Ying Shan, and Ruimao Zhang. Story3d-agent: Exploring 3d storytelling visualization with large language models. arXiv preprint arXiv:2408.11801, 2024. 3

    2024 arXiv

  17. [25]

    Vbench: Comprehensive bench- mark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive bench- mark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  18. [26]

    Videobooth: Diffusion-based video generation with image prompts

    Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu. Videobooth: Diffusion-based video generation with image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6689– 6700, 202...

    2024

  19. [27]

    Text2video-zero: Text- toimage diffusion models are zero-shot video generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text- toimage diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439, 2023. 2

    2023 arXiv

  20. [28]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 4

    2013 arXiv

  21. [29]

    Adam: A method for stochastic opti- mization

    Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980, 2014. 1

    2014 arXiv

  22. [30]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 202...

    2023

  23. [31]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1931–1941, 2023. 2, 3, 7

    1931

  24. [32]

    Open-sora-plan, 2024

    PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, 2024. 3

    2024

  25. [33]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9579–9589, 2024. 6

    2024

  26. [34]

    Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing

    Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Pro- cessing Systems, 36, 2024. 2, 3, 4, 6, 7

    2024

  27. [35]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2, 4

    2023

  28. [36]

    Photomaker: Customizing re- alistic human photos via stacked id embedding

    Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming- Ming Cheng, and Ying Shan. Photomaker: Customizing re- alistic human photos via stacked id embedding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8640–8650, 2024. 2, 3, 4

    2024

  29. [37]

    Flow matching for generative mod- eling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling. arXiv preprint arXiv:2210.02747, 2022. 4

    2022 arXiv

  30. [38]

    Stylecrafter: Enhancing stylized text-to-video gener- ation with style adapter

    Gongye Liu, Menghan Xia, Yong Zhang, Haoxin Chen, Jinbo Xing, Yibo Wang, Xintao Wang, Yujiu Yang, and Ying Shan. Stylecrafter: Enhancing stylized text-to-video gener- ation with style adapter. arXiv preprint arXiv:2312.00330 ,

    arXiv

  31. [39]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485,

    arXiv

  32. [40]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 3, 5, 1

    2023 arXiv

  33. [41]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. 4

    2022 arXiv

  34. [42]

    Videofusion: Decomposed diffusion mod- els for high-quality video generation

    Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion mod- els for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2023

  35. [43]

    Subject- diffusion: Open domain personalized text-to-image genera- tion without test-time fine-tuning

    Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject- diffusion: Open domain personalized text-to-image genera- tion without test-time fine-tuning. In ACM SIGGRAPH 2024 Conference Papers, pages 1–12, 2024. 5

    2024

  36. [44]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  37. [45]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  38. [46]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 4

    2020

  39. [47]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kun- chang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks,

  40. [48]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

    2022

  41. [49]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Pa...

    2015

  42. [50]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  43. [51]

    In- stantbooth: Personalized text-to-image generation without test-time finetuning

    Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. In- stantbooth: Personalized text-to-image generation without test-time finetuning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 8543–8552, 2024. 4

    2024

  44. [52]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  45. [53]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 1

    2010 arXiv

  46. [54]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  47. [55]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 3

    2017

  48. [56]

    Phenaki: Variable length video generation from open domain textual description

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399, 2022. 2

    arXiv

  49. [57]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. 5

    2018 arXiv

  50. [58]

    Modelscope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023. 2, 3

    2023 arXiv

  51. [59]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 6

    2024 arXiv

  52. [60]

    Instantid: Zero-shot identity-preserving generation in seconds

    Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, and Yao Hu. Instantid: Zero-shot identity-preserving generation in seconds. arXiv preprint arXiv:2401.07519, 2024. 2, 3

    2024 arXiv

  53. [61]

    Ms-diffusion: Multi-subject zero-shot im- age personalization with layout guidance

    X Wang, Siming Fu, Qihan Huang, Wanggui He, and Hao Jiang. Ms-diffusion: Multi-subject zero-shot im- age personalization with layout guidance. arXiv preprint arXiv:2406.07209, 2024. 2, 3, 5, 7

    2024 arXiv

  54. [62]

    Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation

    Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943–15953, 2023...

    2023

  55. [63]

    Dreamvideo: Composing your dream videos with customized subject and motion

    Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhi- heng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hong- ming Shan. Dreamvideo: Composing your dream videos with customized subject and motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  56. [64]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565, 2022. 2

    2022 arXiv

  57. [65]

    Fastcomposer: Tuning-free multi- subject image generation with localized attention

    Guangxuan Xiao, Tianwei Yin, William T Freeman, Fr ´edo Durand, and Song Han. Fastcomposer: Tuning-free multi- subject image generation with localized attention. Interna- tional Journal of Computer Vision, pages 1–20, 2024. 3, 4

    2024

  58. [66]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In Eu- ropean Conference on Computer Vision , pages 399–417. Springer, 2025. 4

    2025

  59. [67]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  60. [68]

    I2vgen-xl: High-quality image-to-video 11 synthesis via cascaded diffusion models

    Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. I2vgen-xl: High-quality image-to-video 11 synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04145, 2023. 2, 7

    2023 arXiv

  61. [69]

    Ssr-encoder: Encoding selective subject representation for subject-driven generation

    Yuxuan Zhang, Yiren Song, Jiaming Liu, Rui Wang, Jinpeng Yu, Hao Tang, Huaxia Li, Xu Tang, Yao Hu, Han Pan, et al. Ssr-encoder: Encoding selective subject representation for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  62. [70]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 3

    2024

  63. [71]

    Celebv- hq: A large-scale video facial attributes dataset

    Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv- hq: A large-scale video facial attributes dataset. InEuropean conference on computer vision , pages 650–667. Springer,

  64. [73]

    Introduction of our text-to-video diffusion transformer models

  65. [74]

    Implementation Details of ConceptMaster

  66. [75]

    Discussions on Comparison between Our Data Collec- tion Pipeline and Grounded-SAM

  67. [76]

    More details of Multi-Concept Evaluation Set

  68. [77]

    Comparison Methods Implementation

  69. [78]

    More Discussions on Multi-Concept Embeddings Injec- tion

  70. [79]

    More Discussions on Ablation Study

  71. [80]

    More Qualitative Results Demonstration

  72. [81]

    Introduction of our text-to-video diffusion transformer models We utilize a transformer-based latent diffusion model as the foundational text-to-video (T2V) generation model, as depicted in Fig. 8. Initially, we employ a 3D Varia- tional Autoencoder (3D-V AE) to transform vide...

  73. [82]

    We train ConceptMaster using our proprietary transformer-based text-to-video diffusion models

    Implementation Details of ConceptMaster Implementation Details. We train ConceptMaster using our proprietary transformer-based text-to-video diffusion models. Initially, we employ the CLIP image encoder [45] as the external vision encoder to extract visual features from refere...

  74. [83]

    Discussions on Comparison between Our Data Collection Pipeline and Grounded- SAM Previous studies typically exploit open-set object detec- tion and segmentation methods, represented by Grounded- SAM [30, 40, 47], to extract concepts information in source images or videos. Howe...

    2000

  75. [84]

    We demonstrate the video caption tem- plates for the six different scenarios in Tab

    More details of Multi-Concept Evaluation Set As mentioned in section 4.3, we manually collect reference images and give out suitable captions for these scenarios, in order to eliminate information leakage when extracting con- cepts from videos. We demonstrate the video caption...

  76. [85]

    Comparison Methods Implementation We supplement the implementation details of the com- pared methods for the MCVC task. We compare several open-sourced multi-concept image customization methods, including CustomDiffusion [31], SSR-Encoder [69], IP- Adapter [67] and MS-Diffusio...

  77. [86]

    More Discussions on Multi-Concept Embed- dings Injection We demonstrate more quantitative results between these three different multi-concept embeddings injection meth- ods in Fig. 10. Our key insight is to inject the repre- sented multi-concept embeddings into the diffusion m...

  78. [87]

    More Discussions on Ablation Study We demonstrate more quantitative results of the effective- ness of the Q-Former and DAM modules in Fig. 11. Ini- tially, our ConceptMaster proposes to firstly utilizes a Q- Former network to integrate the dense visual tokens ex- tracted by CL...

  79. [88]

    More Qualitative Results Demonstration Our ConceptMaster could create high-quality and concept- consistent customized videos based on given multiple refer- ence images in diverse scenarios, including but not limited to 1) multiple persons, 2) persons with livings, 3) persons w...

  80. [89]

    We demonstrate more qualitative results including these scenes in Fig

    persons with both livings and stuffs. We demonstrate more qualitative results including these scenes in Fig. 12 and Fig. 13. 3 Diverse Scenarios Caption Templates

  81. [90]

    <ID1> and <ID2> kissing each other in front of a bridge

    Multiple Persons <ID1> and <ID2> hugging each other in front of a bridge. <ID1> and <ID2> kissing each other in front of a bridge. <ID1> and <ID2> walking down a city street. <ID1> and <ID2> dancing on a city street. <ID1> and <ID2> smiling and shaking hands in the office. <ID...

  82. [91]

    <ID1> walking <ID2> in the woods

    Persons with Livings <ID1> walking <ID2> on the beach. <ID1> walking <ID2> in the woods. <ID1> petting <ID2> in the park. <ID1> feeding <ID2> in the garden. <ID1> and <ID2> running on the grass. <ID1> rides <ID2> running on the farm. <ID1> petting <ID2> in the stable. <ID1> ra...

  83. [92]

    <ID1> wearing <ID2> walking along the river

    Persons with Stuffs <ID1> wearing <ID2> walking in the shopping mall. <ID1> wearing <ID2> walking along the river. <ID1> wearing <ID2> running in the stadium. <ID1> wearing <ID2> dancing on the floor. <ID1> rides <ID2> in the desert. <ID1> rides <ID2> on the road. <ID1> is hap...

  84. [93]

    <ID1> and <ID2> walking on the grass

    Multiple Livings <ID1> is playing with <ID2> on the road. <ID1> and <ID2> walking on the grass. <ID1> walking around <ID2> in the desert. <ID1> is playing with <ID2> on the street

  85. [94]

    <ID1> playing with <ID2> in a grassy field

    Livings with Stuffs <ID1> walking around <ID2> in a grassy field. <ID1> playing with <ID2> in a grassy field. <ID1> walking around <ID2> at home. <ID1> is playing with <ID2> at home. <ID1> is walking around <ID2> on the road. <ID1> is staying besides <ID2> in the snow. <ID1> i...

  86. [95]

    A man rides a motorcycle in the desert

    Persons with both Livings and Stuffs <ID1> walking <ID2> in front of <ID3> on the street. <ID1> and <ID2> walking around <ID3> in the snow. <ID1> and <ID2> walking around <ID3> on the road. <ID1> is happily playing <ID2> and <ID3> surrounds in the garden. <ID1> taking <ID2> an...

  87. [2022]

    3, 6 12 ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning Supplementary Material We provide the following contents in supplementary mate- rials:

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.