REVIEW 4 major objections 6 minor 14 references
CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A short test-time training pass lets a pre-trained video diffusion model adopt a new subject's appearance and a reference video's motion at once, without the artifacts that come from merging separately trained adapters.
desk verdict A practical layer-specific LoRA recipe with a test-time distillation step that works empirically, but the TTT objective's asymmetry needs a loss-wise ablation before the decoupling claim is fully supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is layer-selective LoRA placement combined with teacher-distillation test-time training. A LoRA is a low-rank weight update written as $W = W_0 + \triangle W = W_0 + AB$; here the appearance adapter is attached to spatial layers 2 and 6, and the motion adapter to temporal layers 2 and 5, chosen by a prompt-replacement experiment on AnimateDiff. After combination, the model is fine-tuned for 30 steps with two losses: appearance preservation compares the combined model's noise prediction to the noise that produced a DDIM reference latent from the solo appearance model, and temporal preservation compares a frame-anchored relative motion term $\phi(\epsilon_i) = \sqrt{\beta^2+1}\,\epsilon_i - \beta\,\text{anchor}$ against the solo motion model's reference. This lets the combined model inherit each single concept without full weight merging.
What would settle it
Generate a solo appearance reference and a solo motion reference for a case where the subject and motion are very different, inspect the reference latents for artifacts or prompt leakage, and compare the combined model trained with those latents against a version trained with clean ground-truth references; if deliberately corrupting the teacher latents does not degrade the combined output, the distillation is not actually carrying the appearance and motion information.
Extended reading notes
Core claim
CustomTTT claims that in a text-to-video diffusion U-Net, appearance is controlled mainly by the spatial cross-attention layers at indexes 2 and 6, and motion by the temporal attention layers at indexes 2 and 5. Training the appearance LoRA only on spatial layers 2 and 6 and the motion LoRA only on temporal layers 2 and 5 decouples the two concepts. Directly inserting both LoRAs still leaves artifacts, so the paper adds a test-time training stage: generate reference latents from each solo LoRA model with DDIM, then update the combined LoRAs for 30 steps against an appearance preservation loss (Eq. 4) and a temporal preservation loss (Eq. 6). The resulting 12.12M-parameter adaptation achieves the reported best scores on text-video alignment, appearance similarity, motion similarity, temporal consistency, and user rankings.
Load-bearing premise
Everything rests on the assumption that the individually trained appearance and motion adapters produce clean, faithful reference videos; if those references already contain artifacts or wrong attributes, the test-time training will permanently copy those errors into the combined model.
Editorial extensions
If this is right
- Users can combine any few-image subject with any reference-video motion in a single base model using only 12.12M trainable parameters, fewer than the compared methods use.
- The layer analysis suggests that prompt-conditioned video generation has a sparse control structure, with a couple of spatial layers driving appearance and a couple of temporal layers driving motion, which could guide where future adapters are inserted.
- Teacher distillation from solo LoRA models offers a general recipe for merging any pair of individually trained adapters, not only appearance and motion adapters.
- The 30-step test-time training stage is short enough for single-GPU use, making multi-concept customization practical in a typical research setting.
Reading between the lines
- The layer-importance findings may transfer to other U-Net-based video diffusion models with similar spatial and temporal module order, but that transfer is not tested in the paper; a future experiment could repeat the prompt-injection test on another backbone.
- The distillation targets are assumed faithful: if a solo adapter's reference latents already contain artifacts or prompt leakage, the test-time training will entrench those errors in the combined model, and this teacher fidelity is not evaluated independently.
- The temporal preservation loss, being appearance-anchored, could serve as a general regularizer for motion consistency in other video editing tasks beyond customization.
- Part of the reported gains over baselines may come from the AnimateDiff backbone and dataset construction; re-running on a common backbone and fixed prompt set would isolate the contribution of test-time training itself.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes CustomTTT, a method for jointly customizing appearance and motion in a pre-trained text-to-video diffusion model (AnimateDiff). It first identifies spatial layers 2 and 6 and temporal layers 2 and 5 as important for appearance and motion via prompt-embedding replacement experiments, trains separate LoRA adapters on those layers, and then combines them with a test-time training stage. In this stage, reference latents generated by the solo appearance and motion LoRA models are used as distillation targets for the combined model, with an appearance preservation loss (Eq. 4) and a temporal preservation loss (Eq. 6). Experiments compare against DreamVideo, MotionDirector, DiffDirector, and full-LoRA baselines, reporting improvements in CLIP-T, CLIP-I, temporal consistency, and user-study scores.
Significance. If the results hold, the paper is a useful empirical contribution: it shows that layer-selective LoRA placement plus a short self-distillation refinement can reduce artifacts in multi-concept video customization, and it does so with fewer trainable parameters than the compared baselines. The code link and the ablation studies on layer placement, reference-latent step f, and number of training steps are positive elements, and the central idea is plausible and easy to reproduce. The contribution is empirical rather than theoretical; its significance depends on how robust the reported gains are, which is not yet established by the paper's statistics.
major comments (4)
- [Sec. 3.4, Eqs. (4) and (6)] The TTT objective is asymmetric, and the paper does not ablate its two terms. Eq. (4) trains the combined model against a latent z_f^s sampled from the solo appearance LoRA, whose motion content is whatever that teacher happens to produce for prompt c'; Eq. (6) removes the appearance influence from the motion target via the anchor transform, but nothing in Eq. (4) removes or down-weights the appearance teacher's motion. Since z_f^s and z_f^t are generated independently, the two loss terms can pull the combined LoRA weights in conflicting directions. The reported experiments only compare 'with TTT' against 'w/o TTT' (Figs. 8 and 9); no separate L_ap-only or L_tp-only run, no gradient analysis, and no convergence study is provided. A minimal fix is to ablate each loss separately and to report gradient norms of L_ap with respect to ΔW_t^{2,5} and of L_tp with respect to ΔW_s^{2,6} to demonstrate decoupling.
- [Sec. 3.2 and Figs. 4-5] The layer-importance analysis is qualitative and under-supported. The conclusion that spatial layers 2 and 6 control appearance and temporal layers 2 and 5 control motion rests on a handful of prompt-injection videos; no quantitative criterion (e.g., CLIP-I/CLIP-T or motion similarity under systematic layer combinations) is reported, and the temporal selection is a heuristic ('neighbor temporal layers at i=2,5') because the architecture lacks some modules. The passage 'We conduct many experiments... in the supplementary material' points to material not present in this arXiv version. Please either provide the supplement or quantify the layer-sweep results in the main paper.
- [Table 1 and Sec. 4.2] The main quantitative evidence is not statistically supported. Table 1 reports only point estimates for CLIP-T, CLIP-I, and temporal consistency, with no error bars, no number of evaluation videos per cell, and no significance tests; the user study uses 11 participants and 10 generated videos, and the table lists scores without variance or inter-rater statistics. Because the reported improvements over DiffDirector and Full LoRA are modest for some metrics (e.g., temporal consistency 0.971 vs. 0.978), the claim of consistent superiority needs repeated runs or per-item analysis. Please report means and standard deviations over seeds and a paired significance test for the user study.
- [Sec. 3.4 and Sec. 4.4] Teacher fidelity is not evaluated. The distillation targets are latents generated by the solo appearance and motion LoRA models; if these teachers already contain artifacts, missing attributes, or prompt leakage, the TTT phase will entrench those errors in the combined model. The paper's own limitation statement in Sec. 4.4 acknowledges failures when appearance and motion references differ greatly, but it does not test whether the solo teachers are the source of the failure. Please add an evaluation of the solo-LoRA outputs (e.g., whether each teacher preserves the intended attribute and avoids unintended motion/appearance leakage) before using them as references.
minor comments (6)
- [Fig. 8 caption] The caption contains the typo 'A teedy bear'; it should read 'A teddy bear'.
- [Sec. 4.1] The phrase 'we invert 11 participants' should be 'we invite 11 participants' or 'we recruit 11 participants'.
- [Table 1] The column grouping is confusing because the text says 'We employ three metrics' but the table lists seven columns. The caption should explicitly separate objective metrics (CLIP-T, CLIP-I, Temporal consistency) from user-study metrics (Motion similarity, Appearance similarity, Prompt alignment, Video quality).
- [Eq. (5)] β_anchor is described only as 'the selected frame as the anchor frame'; its value, how it is selected, and how Eq. (5) removes appearance are not specified. Please define β and β_anchor concretely.
- [Sec. 3.4] The notation 'DDIM (x, ϵ(c′; ΔW_s^{2,6}))' introduces x without definition; state that x is the initial noise/latent and how the f-step DDIM sampling is run.
- [Sec. 3.3] The claim that AnimateDiff has no temporal layer at i=4 and no cross-attention at i=3,5 needs a short architecture description or a diagram reference so the layer indexing is self-contained.
Circularity Check
No significant circularity: the TTT targets are fixed teacher-generated latents and the main evaluation is against external baselines.
full rationale
CustomTTT's derivation chain is self-contained. The layer selection in Sec. 3.2 is determined by prompt-injection experiments on AnimateDiff, and the LoRA training uses the standard diffusion denoising loss (Eq. 1). In Sec. 3.4, the test-time training objective is a form of self-distillation: the targets z_f^s and z_f^t are generated once by the individually trained LoRA models, then fixed; Eq. 4 and Eq. 6 train the combined LoRAs to denoise those teacher-generated latents. These targets are not functions of the combined model's output, so the losses do not reduce to reproducing the input by construction. The central evaluation is against external baselines (DreamVideo, MotionDirector, DiffDirector, Full LoRA) using CLIP scores, temporal consistency, motion/appearance similarity, and a user study, which provides independent evidence. The paper contains self-citations (e.g., Liu et al. 2024a in Sec. 2.3, Yuan et al. 2023, Zhang et al. 2023), but none is load-bearing for the method or the evaluation. The acknowledged limitation in Sec. 4.4—that large appearance/motion reference differences hurt results—points to an empirical robustness concern about teacher fidelity, not a logical circularity. No fitted parameter is relabeled as a prediction, and no uniqueness claim is imported from the authors' prior work. Therefore no circular step can be quoted.
Assumptions & free parameters
free parameters (7)
- Appearance LoRA placement (spatial layers 2,6) =
layers 2 and 6
- Motion LoRA placement (temporal layers 2,5) =
layers 2 and 5
- Reference latent sampling step f =
5
- Test-time training steps =
30
- LoRA rank =
32
- Learning rates =
1e-5 spatial, 5e-5 temporal, 1e-6 TTT
- Temporal preservation loss parameters beta and beta_anchor =
not reported
assumptions (5)
- domain assumption AnimateDiff's spatial layers 2 and 6 and temporal layers 2 and 5 are the causally important layers for appearance and motion customization.
- domain assumption The pretrained text encoder and the rest of the video diffusion backbone are fixed and unaffected by the LoRA placement choices.
- domain assumption Single-LoRA generated latents are valid distillation targets for the combined model.
- domain assumption The temporal debiased transformation phi removes appearance information so the temporal loss does not interfere with appearance.
- domain assumption CLIP-T, CLIP-I, temporal consistency, and the user-study ratings measure customization quality.
Cite this review
Pith. "Pith review of CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training." pith.science (2026). https://pith.science/paper/CC6VPKJB
@misc{pith2026241215646,
author = {Pith},
title = {Pith review of: CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training},
year = {2026},
howpublished = {\url{https://pith.science/paper/CC6VPKJB}},
note = {Machine review of arXiv:2412.15646}
}
read the original abstract
Benefiting from large-scale pre-training of text-video pairs, current text-to-video (T2V) diffusion models can generate high-quality videos from the text description. Besides, given some reference images or videos, the parameter-efficient fine-tuning method, i.e. LoRA, can generate high-quality customized concepts, e.g., the specific subject or the motions from a reference video. However, combining the trained multiple concepts from different references into a single network shows obvious artifacts. To this end, we propose CustomTTT, where we can joint custom the appearance and the motion of the given video easily. In detail, we first analyze the prompt influence in the current video diffusion model and find the LoRAs are only needed for the specific layers for appearance and motion customization. Besides, since each LoRA is trained individually, we propose a novel test-time training technique to update parameters after combination utilizing the trained customized models. We conduct detailed experiments to verify the effectiveness of the proposed methods. Our method outperforms several state-of-the-art works in both qualitative and quantitative evaluations.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[3]
arXiv preprint arXiv:2307.04725
Animatediff: Ani- mate your personalized text-to-image diffusion models with- out specific tuning. arXiv preprint arXiv:2307.04725. He, K.; Zhang, X.; Ren, S.; and Sun, J
-
[4]
arXiv preprint arXiv:2207.12598
Classifier-free diffusion guid- ance. arXiv preprint arXiv:2207.12598. Hu, E. J.; Shen, Y .; Wallis, P.; Allen-Zhu, Z.; Li, Y .; Wang, S.; Wang, L.; and Chen, W
-
[6]
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1931–1941
Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1931–1941. Liu, W.; Shen, X.; Li, H.; Bi, X.; Liu, B.; Pun, C.-M.; and Cun, X. 2024a. Depth-aware Test-Time Training for Zero-shot Video Object Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vis...
work page 1931
-
[11]
arXiv preprint arXiv:2403.20193
Motion Inversion for Video Cus- tomization. arXiv preprint arXiv:2403.20193. Wang, R.; Sun, Y .; Gandelsman, Y .; Chen, X.; Efros, A. A.; and Wang, X
-
[12]
arXiv preprint arXiv:2307.05014
Test-time training on video streams. arXiv preprint arXiv:2307.05014. Wei, Y .; Zhang, S.; Qing, Z.; Yuan, H.; Liu, Z.; Liu, Y .; Zhang, Y .; Zhou, J.; and Shan, H
-
[13]
arXiv preprint arXiv:2408.06072
CogVideoX: Text-to-Video Diffusion Models with An Ex- pert Transformer. arXiv preprint arXiv:2408.06072. Yu, L.; Yu, B.; Yu, H.; Huang, F.; and Li, Y
-
[14]
arXiv preprint arXiv:2306.00926
Inserting Anybody in Diffusion Models via Celeb Basis. arXiv preprint arXiv:2306.00926. Zhang, W.; Cun, X.; Wang, X.; Zhang, Y .; Shen, X.; Guo, Y .; Shan, Y .; and Wang, F
-
[2012]
arXiv preprint arXiv:1212.0402
UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402. Sterling, S
Show all 14 references
-
[2017]
arXiv preprint arXiv:1704.00675
The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675. Ren, Y .; Zhou, Y .; Yang, J.; Shi, J.; Liu, D.; Liu, F.; Kwon, M.; and Shrivastava, A
2017 arXiv
-
[2020]
arXiv preprint arXiv:2006.10726
Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726. Wang, L.; Shen, G.; Liang, Y .; Tao, X.; Wan, P.; Zhang, D.; Li, Y .; and Chen, Y
2006 arXiv
-
[2021]
arXiv preprint arXiv:2106.09685
Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Jiang, Y .; Wu, T.; Yang, S.; Si, C.; Lin, D.; Qiao, Y .; Loy, C. C.; and Liu, Z
-
[2022]
arXiv preprint arXiv:2208.01618
An image is worth one word: Personalizing text-to-image generation us- ing textual inversion. arXiv preprint arXiv:2208.01618. Gandelsman, Y .; Sun, Y .; Chen, X.; and Efros, A
-
[2023]
arXiv preprint arXiv:2310.19512
Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512. Chen, H.; Zhang, Y .; Cun, X.; Xia, M.; Wang, X.; Weng, C.; and Shan, Y . 2024a. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Pro-...
-
[2024]
arXiv preprint arXiv:2402.14780
Customize-a-video: One- shot motion customization of text-to-video diffusion mod- els. arXiv preprint arXiv:2402.14780. Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Om- mer, B
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.